Techniques for automated programming of robot tasks using language models

Abstract

One embodiment of a method for generating program code to control a robot includes receiving user input specifying a task to be performed by the robot, processing the user input via a first machine learning model to generate a plurality of subtasks for performing the task, and for each subtask included in the plurality of subtasks, processing the subtask via a second machine learning model to generate program code for controlling the robot to perform the subtask.

What is claimed is: 
     
         1 . A computer-implemented method for generating program code to control a robot, the method comprising:
 receiving user input specifying a task to be performed by the robot;   processing the user input via a first machine learning model to generate a plurality of subtasks for performing the task; and   for each subtask included in the plurality of subtasks, processing the subtask via a second machine learning model to generate program code for controlling the robot to perform the subtask.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 performing one or more operations to simulate the robot performing each subtask included in the plurality of subtasks based on the program code for controlling the robot to perform the subtask; and   in response to one or more errors when simulating the robot to perform a first subtask included in the plurality of subtasks, processing the first subtask and the one or more errors via the second machine learning model to generate additional program code for controlling the robot to perform the first subtask.   
     
     
         3 . The computer-implemented method of  claim 1 , wherein the user input is a natural language input, and each of the first machine learning model and the second machine learning model is a language model. 
     
     
         4 . The computer-implemented method of  claim 1 , further comprising, for each subtask included in the plurality of subtasks, selecting one or more examples of program code associated with the subtask from a plurality of examples of program code, wherein processing the subtask via the second machine learning model comprises inputting the subtask and the one or more examples of program code associated with the subtask into the second machine learning model. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein processing the user input via the first machine learning model comprises inputting, into the first machine learning model, at least one of a role of the first machine learning model to generate the plurality of subtasks, one or more rules for output of the first machine learning model, one or more parameters associated with the robot, or one or more parameters associated with the task. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein processing the subtask via the second machine learning model comprises inputting, into the second machine learning model, at least one of a role of the second machine learning model to generate program code, one or more rules for output of the second machine learning model, one or more parameters associated with the robot, one or more parameters associated with the task, one or more definitions associated with a programming language associated with the program code, or one or more examples of program code. 
     
     
         7 . The computer-implemented method of  claim 1 , wherein processing the subtask via the second machine learning model further comprises inputting, into the second machine learning model, at least one of one or more previous outputs of the second machine learning model or one or more errors from one or more simulations of the robot performing the subtask. 
     
     
         8 . The computer-implemented method of  claim 1 , further comprising performing one or more operations to verify that the program code for controlling the robot to perform the subtask is program code. 
     
     
         9 . The computer-implemented method of  claim 1 , wherein the first machine learning model is the second machine learning model. 
     
     
         10 . The computer-implemented method of  claim 1 , further comprising causing the robot to move based on the program code for each subtask. 
     
     
         11 . One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the steps of:
 receiving user input specifying a task to be performed by the robot;   processing the user input via a first machine learning model to generate a plurality of subtasks for performing the task; and   for each subtask included in the plurality of subtasks, processing the subtask via a second machine learning model to generate program code for controlling the robot to perform the subtask.   
     
     
         12 . The one or more non-transitory computer-readable media of  claim 11 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the steps of:
 performing one or more operations to simulate the robot performing each subtask included in the plurality of subtasks based on the program code for controlling the robot to perform the subtask; and   in response to one or more errors when simulating the robot to perform a first subtask included in the plurality of subtasks, processing the first subtask and the one or more errors via the second machine learning model to generate additional program code for controlling the robot to perform the first subtask.   
     
     
         13 . The one or more non-transitory computer-readable media of  claim 12 , wherein the one or more errors include at least one of a run-time exception, a syntax error, a format error, a collision, or a failure to achieve a final state associated with the task. 
     
     
         14 . The one or more non-transitory computer-readable media of  claim 11 , wherein each of the first machine learning model and the second machine learning model comprises a large language model (LLM). 
     
     
         15 . The one or more non-transitory computer-readable media of  claim 11 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of, for each subtask included in the plurality of subtasks, selecting one or more examples of program code associated with the subtask from a plurality of examples of program code, wherein processing the subtask via the second machine learning model comprises inputting the subtask and the one or more examples of program code associated with the subtask into the second machine learning model. 
     
     
         16 . The one or more non-transitory computer-readable media of  claim 11 , wherein processing the user input via the first machine learning model comprises inputting, into the first machine learning model, at least one of a role of the first machine learning model to generate the plurality of subtasks, one or more rules for output of the first machine learning model, one or more parameters associated with the robot, or one or more parameters associated with the task. 
     
     
         17 . The one or more non-transitory computer-readable media of  claim 11 , wherein processing the subtask via the second machine learning model comprises inputting, into the second machine learning model, at least one of a role of the second machine learning model to generate program code, one or more rules for output of the second machine learning model, one or more parameters associated with the robot, one or more parameters associated with the task, one or more definitions associated with a programming language associated with the program code, or one or more examples of program code. 
     
     
         18 . The one or more non-transitory computer-readable media of  claim 11 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of selecting the one or more examples of program code based on a behavior label associated with the subtask. 
     
     
         19 . The one or more non-transitory computer-readable media of  claim 11 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of causing the robot to move based on the program code for each subtask. 
     
     
         20 . A system, comprising:
 one or more memories storing instructions; and   one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
 receive user input specifying a task to be performed by the robot, 
 process the user input via a first machine learning model to generate a plurality of subtasks for performing the task, and 
 for each subtask included in the plurality of subtasks, process the subtask via a second machine learning model to generate program code for controlling the robot to perform the subtask.