Methods for establishing and utilizing sensorimotor programs

Abstract

A method for establishing sensorimotor programs includes specifying a concept relationship that relates a first concept to a second concept and establishes the second concept as higher-order than the first concept; training a first sensorimotor program to accomplish the first concept using a set of primitive actions; and training a second sensorimotor program to accomplish the second concept using the first sensorimotor program and the set of primitive actions.

We claim: 
     
         1 . A method for establishing sensorimotor programs, comprising:
 determining a first environment that represents a first concept; and   training a first sensorimotor program to accomplish the first concept by interacting with the first environment, comprising training the first sensorimotor program using a first reward function that rewards the first sensorimotor program when the first sensorimotor program successfully accomplishes the first concept and correctly returns a value indicating that the first sensorimotor program has successfully accomplished the first concept.   
     
     
         2 . The method of  claim 1 , wherein the first concept is a bring-about concept; wherein the first reward function rewards when the first sensorimotor program successfully brings about the first concept and correctly returns a value indicating that the first sensorimotor program has successfully brought about the first concept. 
     
     
         3 . The method of  claim 1 , wherein the first concept is a classification concept; wherein the first reward function that rewards when the first sensorimotor program correctly returns a value consistent with the presence or non-presence of the first concept in the first environment. 
     
     
         4 . The method of  claim 1 , wherein training the first sensorimotor program to accomplish the first concept further comprises training the first sensorimotor program using a second reward function different from the first reward function that rewards when the first sensorimotor program successfully accomplishes the first concept but fails to return a value indicating that the first sensorimotor program has successfully accomplished the first concept. 
     
     
         5 . The method of  claim 4 , wherein the second reward function is a shaping reward function. 
     
     
         6 . The method of  claim 1 , wherein training the first sensorimotor program to accomplish the first concept comprises using a set of primitive actions to interact with the first environment. 
     
     
         7 . The method of  claim 7 , wherein using the set of primitive actions to interact with the first environment comprises pushing an object of the first environment. 
     
     
         8 . The method of  claim 7 , further comprising executing the trained first sensorimotor program on a robotic arm system comprising a robotic arm actuator, wherein an action of the set of primitive actions actuates the robotic arm actuator. 
     
     
         9 . A method for establishing sensorimotor programs, comprising:
 generating a first plurality of environments that represents a first concept, wherein the first concept is a bring-about concept; and   training a first sensorimotor program to accomplish the first concept using a reward function in each environment of the first plurality, wherein the first sensorimotor program executes actions of a set of primitive actions to accomplish the first concept.   
     
     
         10 . The method of  claim 9 , wherein the first plurality of environments is generated based on recurring content that enables re-use of learned concepts. 
     
     
         9 . method of  claim 9 , wherein each environment of the first plurality of environments is associated with a dynamics model that collectively simulates the actions executed by the first sensorimotor program. 
     
     
         12 . The method of  claim 9 , wherein generating the first plurality of environments comprises generating a superset of environments and filtering the superset of environments to determine the first plurality of environments. 
     
     
         13 . The method of  claim 9 , further comprising:
 generating a second plurality of environments that represent a second concept higher-order than the first concept; and   training, using the second plurality of environments, a second sensorimotor program to accomplish the second concept using the first sensorimotor program.   
     
     
         14 . The method of  claim 13 , wherein the second sensorimotor program is trained using the set of primitive actions, wherein the second sensorimotor program calls the first sensorimotor program as an additional action. 
     
     
         15 . The method of  claim 13 , wherein the second sensorimotor program calls the first sensorimotor program as an observation. 
     
     
         16 . The method of  claim 9 , wherein the first sensorimotor program is trained based on a set of actionable lower-level sensorimotor programs associated with different respective bring-about concepts and a set of conceptual lower-level sensorimotor programs associated with different respective classification concepts. 
     
     
         17 . The method of  claim 16 , wherein the set of actionable lower-level sensorimotor programs and the set of conceptual lower-level sensorimotor programs are trained before the first sensorimotor program is trained. 
     
     
         18 . The method of  claim 16 , wherein training the first sensorimotor program comprises automatically determining which programs of the actionable lower-level sensorimotor programs and the conceptual lower-level sensorimotor programs enable the first sensorimotor program to accomplish the first concept. 
     
     
         19 . The method of  claim 9 , wherein the first sensorimotor program is represented by a recurrent neural network. 
     
     
         20 . The method of  claim 9 , wherein the first sensorimotor program is trained using natural policy optimization.