Action selection for reinforcement learning using a manager neural network that generates goal vectors defining agent objectives

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a system configured to select actions to be performed by an agent that interacts with an environment. The system comprises a manager neural network subsystem and a worker neural network subsystem. The manager subsystem is configured to, at each of the multiple time steps, generate a final goal vector for the time step. The worker subsystem is configured to, at each of multiple time steps, use the final goal vector generated by the manager subsystem to generate a respective action score for each action in a predetermined set of actions.

What is claimed is: 
     
         1 . A system for selecting actions to be performed by an agent that interacts with an environment by performing actions from a predetermined set of actions, the system comprising one or more computers and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to implement:
 a manager neural network subsystem that is configured to, at each of a plurality of time steps:
 generate a latent representation, in a latent space, of a current state of the environment at the time step; 
 generate, based at least in part on the latent representation of the current state of the environment at the time step, a goal vector that defines, in the latent state space, an objective to be accomplished as a result of actions performed by the agent in the environment; and 
   a worker neural network subsystem that is configured to, at each of the plurality of time steps:
 generate a respective action score for each action in the predetermined set of actions based at least in part on the goal vector for the time step; and 
 select an action from the predetermined set of actions to be performed by the agent at the time step using the action scores; and 
   a training subsystem that is configured to perform operations comprising:
 determining a respective reward for each time step of the plurality of time steps, comprising, for one or more time steps:
 determining the reward for the time step based at least in part on a difference in direction between: (i) a vector representing a change in the latent representation of the state of the environment from a preceding time step to the time step, and (ii) the goal vector for the preceding time step; and 
 
 training the worker neural network subsystem on the rewards using reinforcement learning techniques. 
   
     
     
         2 . The system of  claim 1 , wherein at each of the plurality of time steps, the manager neural network subsystem is further configured to:
 update the goal vector for the time step by pooling the goal vector for the time step with goals vectors for one or more preceding time steps.   
     
     
         3 . The system of  claim 1 , wherein at each of the plurality of time steps, generating the goal vector, comprises:
 processing the latent representation using a goal recurrent neural network, wherein the goal recurrent neural network is configured to receive the latent representation and to process the latent representation in accordance with a hidden state of the goal recurrent neural network to generate the goal vector and to update the hidden state of the goal recurrent neural network.   
     
     
         4 . The system of  claim 1 , wherein at each of the plurality of time steps, generating the respective action score for each action in the predetermined set of actions comprises:
 generating a respective action embedding vector in an embedding space for each action in the predetermined set of actions;   projecting the goal vector for the time step to the embedding space to generate a goal embedding vector; and   modulating the respective action embedding vector for each action by the goal embedding vector to generate the respective action score for each action in the predetermined set of actions.   
     
     
         5 . The system of  claim 1 , wherein selecting the action comprises selecting the action having a highest action score. 
     
     
         6 . The system of  claim 4 , wherein generating the respective action embedding vector in the embedding space for each action in the predetermined set of actions comprises:
 processing a representation of the current state of the environment using an action score recurrent neural network, in accordance with a hidden state of the action score recurrent neural network, to generate the action embedding vectors and to update the hidden state of the action score recurrent neural network.   
     
     
         7 . The system of  claim 4 , wherein the goal vector has a higher dimensionality than the goal embedding vector. 
     
     
         8 . The system of  claim 7 , wherein the dimensionality of the goal vector is at least ten times higher than the dimensionality of the goal embedding vector. 
     
     
         9 . The system of  claim 1 , wherein determining the respective reward for each time step of the plurality of time steps comprises, for one or more time steps:
 receiving an external reward for the time step as a result of the agent performing selected actions; and   determining the reward for the time step based at least in part on the external reward for the time step.   
     
     
         10 . The system of  claim 1 , wherein training the worker neural network subsystem on the rewards using reinforcement learning techniques comprises:
 training the worker neural network subsystem to generate action scores that maximize a time discounted combination of rewards.   
     
     
         11 . The system of  claim 9 , wherein the operations performed by the training subsystem further comprise:
 training the manager neural network subsystem to generate goal vectors that result in action scores that encourage selection of actions that increase the external rewards received as a result of the agent performing the selected actions.   
     
     
         12 . The system of  claim 1 , wherein generating the latent representation, in the latent space, of the current state of the environment at the time step comprises:
 processing an observation characterizing the current state of the environment using a convolutional neural network.   
     
     
         13 . The system of  claim 3 , wherein the goal recurrent neural network is a dilated long short-term memory (LSTM) neural network, wherein the dilated LSTM neural network is configured to maintain an internal state that is partitioned into r sub-states, wherein r is an integer greater than one, and wherein the dilated LSTM neural network is configured to, at each time step in the plurality of time steps:
 receive a network input for the time step;   select a sub-state from the r sub-states; and   process current values of the selected sub-state and the network input for the time step using an LSTM neural network to update the current values of the selected sub-state and to generate a network output for the time step in accordance with current values of a set of LSTM network parameters.   
     
     
         14 . The system of  claim 13 , wherein the dilated LSTM neural network is further configured to, for each of the time steps:
 pool the network output for the time step and the network outputs for up to a predetermined number of preceding time steps to generate a final network output for the time step.   
     
     
         15 . The system of  claim 14 , wherein pooling the network outputs comprises summing the network outputs. 
     
     
         16 . The system of  claim 13 , wherein the time steps in the plurality of time steps are indexed starting from 1 for the first time step in the plurality of time steps to T for the last time step in the plurality of time steps, wherein each sub-state is assigned an index ranging from 1 to r, and wherein selecting a sub-state from the r sub-states comprises:
 selecting the sub-state having an index that is equal to the index of the time step modulo r.   
     
     
         17 . The system of  claim 13 , wherein the LSTM neural network comprises a plurality of LSTM layers. 
     
     
         18 . The system of  claim 13 , wherein processing current values of the selected sub-state and the network input for the time step using an LSTM neural network to update the current values of the selected sub-state and to generate a network output for the time step in accordance with current values of a set of LSTM network parameters comprises:
 setting an internal state of the LSTM neural network to the current values of the selected sub-state for the processing of the network input at the time step.   
     
     
         19 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting actions to be performed by an agent that interacts with an environment by performing actions from a predetermined set of actions, the operations comprising:
 at each of a plurality of time steps:
 generating, by a manager neural network system, a latent representation, in a latent space, of a current state of the environment at the time step; 
 generating, by the manager neural network system, based at least in part on the latent representation of the current state of the environment at the time step, a goal vector that defines, in the latent state space, an objective to be accomplished as a result of actions performed by the agent in the environment; 
 generating, by a worker neural network system, a respective action score for each action in the predetermined set of actions based at least in part on the goal vector for the time step; and 
 selecting, by a worker neural network system, an action from the predetermined set of actions to be performed by the agent at the time step using the action scores; and 
   training the manager neural network system and the worker neural network system, comprising:
 determining a respective reward for each time step of the plurality of time steps, comprising, for one or more time steps:
 determining the reward for the time step based at least in part on a difference in direction between: (i) a vector representing a change in the latent representation of the state of the environment from a preceding time step to the time step, and (ii) the goal vector for the preceding time step; and 
 
 training the worker neural network system on the rewards using reinforcement learning techniques. 
   
     
     
         20 . A method performed by one or more data processing apparatus for selecting actions to be performed by an agent that interacts with an environment by performing actions from a predetermined set of actions, the method comprising:
 at each of a plurality of time steps:
 generating, by a manager neural network system, a latent representation, in a latent space, of a current state of the environment at the time step; 
 generating, by the manager neural network system, based at least in part on the latent representation of the current state of the environment at the time step, a goal vector that defines, in the latent state space, an objective to be accomplished as a result of actions performed by the agent in the environment; 
 generating, by a worker neural network system, a respective action score for each action in the predetermined set of actions based at least in part on the goal vector for the time step; and 
 selecting, by a worker neural network system, an action from the predetermined set of actions to be performed by the agent at the time step using the action scores; and 
   training the manager neural network system and the worker neural network system, comprising:
 determining a respective reward for each time step of the plurality of time steps, comprising, for one or more time steps:
 determining the reward for the time step based at least in part on a difference in direction between: (i) a vector representing a change in the latent representation of the state of the environment from a preceding time step to the time step, and (ii) the goal vector for the preceding time step; and 
 
 training the worker neural network system on the rewards using reinforcement learning techniques. 
   
     
     
         21 . A method performed by one or more data processing apparatus for selecting actions to be performed by an agent that interacts with an environment by performing actions from a predetermined set of actions, the method comprising:
 at each of a plurality of time steps:
 generating, by a manager neural network system, a latent representation, in a latent space, of a current state of the environment at the time step; 
 generating, by the manager neural network system, based at least in part on the latent representation of the current state of the environment at the time step, a goal vector that defines, in the latent state space, an objective to be accomplished as a result of actions performed by the agent in the environment; 
 generating, by a worker neural network system, a respective action score for each action in the predetermined set of actions based at least in part on the goal vector for the time step; and 
 selecting, by a worker neural network system, an action from the predetermined set of actions to be performed by the agent at the time step using the action scores; 
   wherein the manager neural network system and the worker neural network system have been trained by operations comprising:
 determining a respective reward for each training time step of a plurality of training time steps, comprising, for one or more training time steps:
 determining the reward for the training time step based at least in part on a difference in direction between: (i) a vector representing a change in a latent representation of the state of the environment from a preceding training time step to the training time step, and (ii) a goal vector for the preceding time step; and 
 
 training the worker neural network system on the rewards using reinforcement learning techniques. 
   
     
     
         22 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting actions to be performed by an agent that interacts with an environment by performing actions from a predetermined set of actions, the operations comprising:
 at each of a plurality of time steps:
 generating, by a manager neural network system, a latent representation, in a latent space, of a current state of the environment at the time step; 
 generating, by the manager neural network system, based at least in part on the latent representation of the current state of the environment at the time step, a goal vector that defines, in the latent state space, an objective to be accomplished as a result of actions performed by the agent in the environment; 
 generating, by a worker neural network system, a respective action score for each action in the predetermined set of actions based at least in part on the goal vector for the time step; and 
 selecting, by a worker neural network system, an action from the predetermined set of actions to be performed by the agent at the time step using the action scores; 
   wherein the manager neural network system and the worker neural network system have been trained by operations comprising:
 determining a respective reward for each training time step of a plurality of training time steps, comprising, for one or more training time steps:
 determining the reward for the training time step based at least in part on a difference in direction between: (i) a vector representing a change in a latent representation of the state of the environment from a preceding training time step to the training time step, and (ii) a goal vector for the preceding time step; and 
 
 training the worker neural network system on the rewards using reinforcement learning techniques.