Robot systems, methods, control modules, and computer program products that leverage large language models

Abstract

Robot control systems, methods, control modules and computer program products that leverage one or more large language model(s) (LLMs) in order to achieve at least some degree of autonomy are described. Robot control parameters and/or instructions may advantageously be specified in natural language (NL) and communicated with the LLM via a recursive sequence of NL prompts or queries. Corresponding NL responses from the LLM may then be converted into robot control parameters and/or instructions. In this way, an LLM may be leveraged by the robot control system to enhance the autonomy of various operations and/or functions, including without limitation task planning, motion planning, human interaction, and/or reasoning about the environment.

What is claimed is: 
     
       1. A robot system comprising:
 a robot body; 
 at least one sensor to capture sensor data representing information about an environment of the robot body; 
 at least one processor; and 
 at least one non-transitory processor-readable medium communicatively coupled to the at least one processor and storing processor-executable instructions and/or data that, when executed by the at least one processor, cause the at least one processor to: 
 provide a natural language (NL) query to a large language model (LLM) module, the NL query including information about an environment of the robot body, an NL description of a work objective, an NL description of an instruction set which includes a plurality of instructions each selectively executable by the robot system, an NL request for a task plan, and an NL indication of a task plan step threshold corresponding to a maximum number of steps in the task plan; 
 receive the task plan from the LLM module, the task plan expressed in NL and including a number of steps which does not exceed the task plan step threshold; 
 for each step of the task plan which does not correspond to a single instruction in the instruction set:
 provide an NL breakdown query for a respective component plan to the LLM module, the NL breakdown query including an NL request to break down the step of the task plan into a component plan including a plurality of component steps; 
 receive each component plan from the LLM module, each component plan expressed in NL; and 
 incorporate each component plan into the task plan; 
 
 and 
 cause the robot system to execute the task plan. 
 
     
     
       2. The robot system of  claim 1 , wherein for each step of the task plan which does not correspond to a single instruction in the instruction set:
 the NL breakdown query includes an NL indication of a component plan step threshold indicating a maximum number of steps in the component plan; and 
 the component plan includes a number of steps which does not exceed the component plan step threshold. 
 
     
     
       3. The robot system of  claim 2 , wherein the processor-executable instructions and/or data, when executed by the at least one processor, further cause the at least one processor to, for each step of a component plan which does not correspond to a single instruction in the instruction set:
 provide a further NL breakdown query for a respective further component plan to the LLM module, the further NL query including an NL request to break down the step of the component plan into a further component plan including a plurality of component steps; 
 receive each further component plan from the LLM module, each further component plan expressed in NL; and 
 incorporate each further component plan into the task plan. 
 
     
     
       4. The robot system of  claim 2 , wherein the task plan step threshold is equal to the component plan step threshold. 
     
     
       5. The robot system of  claim 1 , wherein the processor-executable instructions and/or data, when executed by the at least one processor, further cause the at least one processor to determine, for each step of the task plan, whether the step corresponds to a single instruction in the instruction set. 
     
     
       6. The robot system of  claim 1 , wherein:
 each step of the task plan which corresponds to a single instruction in the instruction set indicates the respective single instruction in natural language; 
 the processor-executable instructions and/or data, when executed by the at least one processor, further cause the at least one processor to, after incorporating each component plan into the task plan and before executing the task plan, generate a robot-language task plan based on the task plan, the robot-language task plan comprising a sequence of robot-language instructions in the instruction set which when executed by the at least one processor cause the robot system to perform each step of the task plan; and 
 the processor-executable instructions and/or data that, when executed by the at least one processor, cause the robot system to execute the task plan, cause the at least one processor to execute the robot-language task plan to cause the robot system to execute the task plan. 
 
     
     
       7. The robot system of  claim 6 , wherein the processor-executable instructions and/or data that, when executed by the at least one processor, cause the at least one processor to generate the robot-language task plan, cause the at least one processor to execute a robot-language conversion module which converts the respective single instructions indicated in natural language to at least one reusable work primitive in the instruction set executable by the robot system. 
     
     
       8. The robot system of  claim 1 , wherein the processor-executable instructions and/or data, when executed by the at least one processor, further cause the at least one processor to generate the NL description of the instruction set, wherein generating the NL description of the instruction set includes executing a robot-language conversion module which generates an NL description of each instruction in the instruction set as expressed in robot-language executable by the robot system. 
     
     
       9. A robot system comprising:
 a robot body; 
 at least one sensor to capture sensor data representing information about an environment of the robot body; 
 at least one processor; and 
 at least one non-transitory processor-readable medium communicatively coupled to the at least one processor and storing processor-executable instructions and/or data that, when executed by the at least one processor, cause the at least one processor to: 
 provide a natural language (NL) query to a large language model (LLM) module, the NL query including information about an environment of the robot body, an NL description of a work objective, an NL description of an instruction set which includes a plurality of instructions each selectively executable by the robot system, an NL request for a task plan, and an NL indication of a task plan step threshold indicating a maximum number of steps in the task plan; 
 receive the task plan from the LLM module, the task plan expressed in NL and including a number of steps which does not exceed the task plan step threshold; 
 for each step of the task plan which does not correspond to a single instruction in the instruction set, until each step of the task plan does correspond to a single instruction in the instruction set:
 provide an NL breakdown query to the LLM module, the NL breakdown query including an NL request to break down the step of the task plan into a plurality of component steps; 
 receive the plurality of component steps from the LLM module expressed in NL; and 
 incorporate the plurality of component steps from the LLM module into the task plan as steps; 
 
 and 
 cause the robot system to execute the task plan. 
 
     
     
       10. The robot system of  claim 9 , wherein the processor-executable instructions and/or data, when executed by the at least one processor, further cause the at least one processor to, for each step of the task plan, determine whether the step corresponds to a single instruction in the instruction set. 
     
     
       11. The robot system of  claim 9 , wherein for each step of the task plan which does not correspond to a single instruction in the instruction set, the processor-executable instructions and/or data that, when executed by the at least one processor, cause the at least one processor to incorporate the plurality of component steps from the LLM module into the task plan, cause the at least one processor to replace the respective step with the plurality of component steps. 
     
     
       12. The robot system of  claim 9 , wherein:
 each step of the task plan which corresponds to a single instruction in the instruction set indicates the respective single instruction in natural language; 
 the processor-executable instructions and/or data, when executed by the at least one processor, further cause the at least one processor to, before causing the robot system to execute the task plan, generate a robot-language task plan based on the task plan, the robot-language task plan comprising a sequence of robot-language instructions in the instruction set which when executed by the at least one processor cause the robot system to perform each step of the task plan; and 
 the processor-executable instructions and/or data that, when executed by the at least one processor, cause the robot system to execute the task plan, cause the at least one processor to execute the robot-language task plan to cause the robot system to execute the task plan. 
 
     
     
       13. The robot system of  claim 12 , wherein the processor-executable instructions and/or data that, when executed by the at least one processor, cause the at least one processor to generate the robot-language task plan, cause the at least one processor to execute a robot-language conversion module which converts the respective single instructions indicated in natural language to at least one reusable work primitive in the instruction set executable by the robot system. 
     
     
       14. A robot system comprising:
 a robot body; 
 at least one sensor to capture sensor data representing information about an environment of the robot body; 
 at least one processor; and 
 at least one non-transitory processor-readable medium communicatively coupled to the at least one processor and storing processor-executable instructions and/or data that, when executed by the at least one processor, cause the at least one processor to: 
 provide a first NL query to a large language model (LLM) module, the first NL query including information about an environment of the robot body, an NL description of a work objective, an NL description of an instruction set which includes a plurality of instructions each selectively executable by the robot system, an NL request for a task plan corresponding to a sequence of instructions from the instruction set, and an NL indication of a task plan instruction threshold for a maximum number of instructions in the sequence; 
 receive an indication from the LLM module that generation of a task plan having a number of instructions which does not exceed the task plan instruction threshold has failed; 
 provide a second NL query to the LLM module, the second NL query including a request to provide the task plan as a set of conceptual steps which does not exceed a task plan step threshold for a maximum number of steps; 
 receive the task plan from the LLM module as the set of conceptual steps expressed in NL; 
 for each step of the set of conceptual steps, until the task plan comprises only instructions specified in the instruction set:
 provide a respective NL breakdown query to the LLM module, the respective NL breakdown query including an NL request to break down the conceptual step of the task plan into a plurality of instructions from the set of instructions; 
 receive either an NL description of the plurality of instructions from the LLM module, or an indication from the LLM module that generation of the plurality of instructions which does not exceed the task plan instruction threshold has failed; 
 when an NL description of the plurality of instructions is received, incorporate the plurality of instructions into the task plan; 
 when an indication from the LLM module that generation of the plurality of instructions which does not exceed the task plan instruction threshold has failed, provide a further NL breakdown query to the LLM module, the further NL breakdown query including a request to provide the conceptual step as a further set of conceptual steps which does not exceed a step threshold for a maximum number of steps; 
 
 and 
 cause the robot system to execute the task plan. 
 
     
     
       15. The robot system of  claim 14 , wherein the processor-executable instructions and/or data that, when executed by the at least one processor, cause the at least one processor to incorporate the plurality of instructions into the task plan, further cause the at least one processor to replace the corresponding step with the plurality of instructions. 
     
     
       16. The robot system of  claim 14 , wherein the processor-executable instructions and/or data, when executed by the at least one processor, further cause the at least one processor to, for each step of the set of conceptual steps:
 in response to providing a further NL breakdown query to the LLM module, receive the further set of conceptual steps; and 
 incorporate the further set of conceptual steps into the task plan. 
 
     
     
       17. The robot system of  claim 16 , wherein for each step of the set of conceptual steps, the processor-executable instructions and/or data that, when executed by the at least one processor, cause the at least one processor to incorporate the further set of conceptual steps into the task plan, further cause the at least one processor to replace the corresponding step with the further set of conceptual steps. 
     
     
       18. The robot system of  claim 14 , wherein the task plan instruction threshold for a maximum number of instructions in the sequence is equal to the task plan step threshold for a maximum number of steps. 
     
     
       19. The robot system of  claim 14 , wherein:
 each instruction in the task plan is expressed in natural language; 
 the processor-executable instructions and/or data, when executed by the at least one processor, further cause the at least one processor to, before executing the task plan, generate a robot-language task plan based on the task plan, the robot-language task plan comprising a sequence of robot-language instructions in the instruction set which when executed by the at least one processor cause the robot system to perform each instruction of the task plan; and 
 the processor-executable instructions and/or data that, when executed by the at least one processor, cause the robot system to execute the task plan, cause the at least one processor to execute the robot-language task plan to cause the robot system to execute the task plan. 
 
     
     
       20. The robot system of  claim 19 , wherein the processor-executable instructions and/or data that, when executed by the at least one processor, cause the at least one processor to generate the robot-language task plan, cause the at least one processor to execute a robot-language conversion module which converts the instructions indicated in natural language to respective reusable work primitives in the instruction set executable by the robot system.