Imagination-based agent neural networks

Abstract

A neural network system is proposed to select actions to be performed by an agent interacting with an environment to perform a task in an attempt to achieve a specified result. The system may include a controller to receive state data and context data, and to output action data. The system may also include an imagination module to receive the state and action data, and to output consequent state data. The system may also include a manager to receive the state data and the context data, and to output route data which defines whether the system is to execute an action or to imagine. The system may also include a memory to store the context data.

What is claimed is: 
     
         1 . (canceled) 
     
     
         2 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for controlling an agent to perform a specified task in an environment, the operations comprising:
 receiving current state data characterizing a current state of the environment;   until determining to act in the environment in response to the current state data, performing iterations of an imagining process to imagine consequences of performing actions in the environment, wherein performing each iteration of the imagining process comprising:
 selecting input state data for the iteration; 
 processing an input comprising at least the input state data for the iteration using a controller neural network configured to process the input context data and to output action data defining an action to be performed by the agent when the environment is in a state defined by the input state data; 
 processing the input state data and the action data using a model neural network configured to process the input state data and the action data to (i) generate predicted state data that characterizes a predicted state that the environment transitions into consequent upon the action defined by the action data being performed while the environment is in the state defined by the input state data and (ii) generate a predicted reward that is a prediction of a reward that would be received consequent upon the action defined by the action data being performed while the environment is in the state defined by the input sate data; and 
 determining whether to (i) act in the environment or (ii) perform another iteration of the imagining process using at least the predicted state data and the predicted reward for the iteration; and 
   after determining to act in the environment, selecting an action to be performed by the agent in response to the current state data based at least on results of a last iteration of the imagining process.   
     
     
         3 . The system of  claim 2 , wherein, for a first iteration of the imagining process, selecting the input state data comprises selecting the current state data as the input state data for the iteration. 
     
     
         4 . The system of  claim 3 , wherein, for any iteration of the imagining process after the first iteration, selecting the input state data comprises selecting the predicted state generated at the preceding iteration of the imagining process as the input state at the iteration. 
     
     
         5 . The system of  claim 3 , wherein, for any iteration of the imagining process after the first iteration, selecting the input state data comprises selecting the input state for the iteration from at least the predicted state generated at the preceding iteration of the imagining process and the current state data. 
     
     
         6 . The system of  claim 2 , wherein determining whether to (i) act in the environment or (ii) perform another iteration of the imagining process comprises:
 processing a manager input generated from at least the predicted state data and the predicted reward for the iteration using a manager neural network configured to process the manager input to generate as output route data that defines whether to (i) act in the environment or (ii) perform another iteration of the imagining process.   
     
     
         7 . The system of  claim 6 , wherein the route data further defines the state data that is selected as the input state data at the another iteration of the imagining process. 
     
     
         8 . The system of  claim 6 , the operations further comprising:
 maintaining context data defining a context for planning actions to be performed by the agent; wherein:   performing each iteration of the imagining process further comprises updating, using at least the predicted state data and the predicted reward for the iteration, the context data; and   the manager input comprises the updated context data.   
     
     
         9 . The system of  claim 8 , wherein updating, using at least the predicted state data and the predicted reward for the iteration, the context data comprises:
 generating, from at least the predicted state data for the iteration, the predicted reward for the iteration, and the context data, an input to a memory neural network; and   processing the input to the memory neural network using the memory neural network, wherein the memory neural network is configured to generate, as output, updated context data.   
     
     
         10 . The system of  claim 8 , the operations further comprising:
 prior to performing any iterations of the imagining process, processing an input comprising the context data prior to performing any iterations of the imagining process using the manager neural network to generate initial route data that defines whether to (i) act in the environment or (ii) perform any iterations of the imagining process; and   performing iterations of the imagining process to imagine consequences of performing actions in the environment only in response to the initial route data defining to perform iterations of the imagining process.   
     
     
         11 . The system of  claim 8 , wherein after determining to act in the environment, selecting an action to be performed by the agent in response to the current state data comprises:
 selecting an action based at least in part on the context data after the last iteration of the imagining process.   
     
     
         12 . The system of  claim 11 , wherein selecting an action based at least in part on the context data after the last iteration of the imagining process comprises:
 processing at least the current state data and the context data after the last iteration of the imagining process using the controller neural network to generate current action data; and   selecting the action that is defined by the current action data.   
     
     
         13 . The system of  claim 8 , the operations further comprising:
 receiving next state data characterizing a next state that the environment transitioned into as a consequence of the agent performing the selected action;   receiving an external reward as a consequence of the agent performing the selected action; and   updating the context data using at least the next state data and the external reward.   
     
     
         14 . A method performed by one or more computers for controlling an agent in an environment, the method comprising:
 receiving current state data characterizing a current state of the environment;   until determining to act in the environment in response to the current state data, performing iterations of an imagining process to imagine consequences of performing actions in the environment, wherein performing each iteration of the imagining process comprising:
 selecting input state data for the iteration; 
 processing an input comprising at least the input state data for the iteration using a controller neural network configured to process the input context data and to output action data defining an action to be performed by the agent when the environment is in a state defined by the input state data; 
 processing the input state data and the action data using a model neural network configured to process the input state data and the action data to (i) generate predicted state data that characterizes a predicted state that the environment transitions into consequent upon the action defined by the action data being performed while the environment is in the state defined by the input state data and (ii) generate a predicted reward that is a prediction of a reward that would be received consequent upon the action defined by the action data being performed while the environment is in the state defined by the input sate data; and 
 determining whether to (i) act in the environment or (ii) perform another iteration of the imagining process using at least the predicted state data and the predicted reward for the iteration; and 
   after determining to act in the environment, selecting an action to be performed by the agent in response to the current state data based at least on results of a last iteration of the imagining process.   
     
     
         15 . The method of  claim 14 , wherein, for a first iteration of the imagining process, selecting the input state data comprises selecting the current state data as the input state data for the iteration. 
     
     
         16 . The method of  claim 15 , wherein, for any iteration of the imagining process after the first iteration, selecting the input state data comprises selecting the predicted state generated at the preceding iteration of the imagining process as the input state at the iteration. 
     
     
         17 . The method of  claim 15 , wherein, for any iteration of the imagining process after the first iteration, selecting the input state data comprises selecting the input state for the iteration from at least the predicted state generated at the preceding iteration of the imagining process and the current state data. 
     
     
         18 . The method of  claim 15 , wherein determining whether to (i) act in the environment or (ii) perform another iteration of the imagining process comprises:
 processing a manager input generated from at least the predicted state data and the predicted reward for the iteration using a manager neural network configured to process the manager input to generate as output route data that defines whether to (i) act in the environment or (ii) perform another iteration of the imagining process.   
     
     
         19 . The method of  claim 18 , wherein the route data further defines the state data that is selected as the input state data at the another iteration of the imagining process. 
     
     
         20 . The method of  claim 19 , further comprising:
 maintaining context data defining a context for planning actions to be performed by the agent; wherein:   performing each iteration of the imagining process further comprises updating, using at least the predicted state data and the predicted reward for the iteration, the context data; and   the manager input comprises the updated context data.   
     
     
         21 . The method of  claim 20 , wherein updating, using at least the predicted state data and the predicted reward for the iteration, the context data comprises:
 generating, from at least the predicted state data for the iteration, the predicted reward for the iteration, and the context data, an input to a memory neural network; and   processing the input to the memory neural network using the memory neural network, wherein the memory neural network is configured to generate, as output, updated context data.   
     
     
         22 . The method of  claim 20 , further comprising:
 prior to performing any iterations of the imagining process, processing an input comprising the context data prior to performing any iterations of the imagining process using the manager neural network to generate initial route data that defines whether to (i) act in the environment or (ii) perform any iterations of the imagining process; and   performing iterations of the imagining process to imagine consequences of performing actions in the environment only in response to the initial route data defining to perform iterations of the imagining process.   
     
     
         23 . The method of  claim 20 , wherein after determining to act in the environment, selecting an action to be performed by the agent in response to the current state data comprises:
 selecting an action based at least in part on the context data after the last iteration of the imagining process.   
     
     
         24 . The method of  claim 23 , wherein selecting an action based at least in part on the context data after the last iteration of the imagining process comprises:
 processing at least the current state data and the context data after the last iteration of the imagining process using the controller neural network to generate current action data; and   selecting the action that is defined by the current action data.   
     
     
         25 . The method of  claim 20 , further comprising:
 receiving next state data characterizing a next state that the environment transitioned into as a consequence of the agent performing the selected action;   receiving an external reward as a consequence of the agent performing the selected action; and   updating the context data using at least the next state data and the external reward.   
     
     
         26 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for controlling an agent in an environment, the operations comprising:
 receiving current state data characterizing a current state of the environment;   until determining to act in the environment in response to the current state data, performing iterations of an imagining process to imagine consequences of performing actions in the environment, wherein performing each iteration of the imagining process comprising:
 selecting input state data for the iteration; 
 processing an input comprising at least the input state data for the iteration using a controller neural network configured to process the input context data and to output action data defining an action to be performed by the agent when the environment is in a state defined by the input state data; 
 processing the input state data and the action data using a model neural network configured to process the input state data and the action data to (i) generate predicted state data that characterizes a predicted state that the environment transitions into consequent upon the action defined by the action data being performed while the environment is in the state defined by the input state data and (ii) generate a predicted reward that is a prediction of a reward that would be received consequent upon the action defined by the action data being performed while the environment is in the state defined by the input sate data; and 
 determining whether to (i) act in the environment or (ii) perform another iteration of the imagining process using at least the predicted state data and the predicted reward for the iteration; and 
   after determining to act in the environment, selecting an action to be performed by the agent in response to the current state data based at least on results of a last iteration of the imagining process.