Jointly learning exploratory and non-exploratory action selection policies

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection neural network that is used to select actions to be performed by an agent interacting with an environment. In one aspect, the method comprises: receiving an observation characterizing a current state of the environment; processing the observation and an exploration importance factor using the action selection neural network to generate an action selection output; selecting an action to be performed by the agent using the action selection output; determining an exploration reward; determining an overall reward based on: (i) the exploration importance factor, and (ii) the exploration reward; and training the action selection neural network using a reinforcement learning technique based on the overall reward.

1 . A method performed by one or more computers for selecting an action to be performed by an agent interacting with an environment, the method comprising:
 receiving an observation characterizing a current state of the environment;   processing: (i) the observation, and (ii) the exploration importance factor, using an action selection neural network to generate an action selection output, wherein:   the action selection neural network implements a plurality of action selection policies indexed by a set of possible exploration importance factors; and   the plurality of action selection policies comprise an exploitative action selection policy and one or more exploratory action selection policies; and   selecting an action to be performed by the agent using the action selection output.   
     
     
         2 . The method of  claim 1 , wherein the exploration importance factor processed by the action selection neural network is a baseline exploration importance factor causing the agent to implement the exploitative action selection policy. 
     
     
         3 . The method of  claim 2 , wherein baseline exploration importance factor has value zero. 
     
     
         4 . The method of  claim 1 , wherein the action selection policies have been trained to optimize an overall reward comprising a task reward and an exploration reward. 
     
     
         5 . The method of  claim 4 , wherein the exploitative action selection policy is an action selection policy that is trained to optimize an overall reward independent of the exploration reward. 
     
     
         6 . The method of  claim 1 , wherein the exploitative action selection policy causes the agent to exploit its current knowledge of the environment. 
     
     
         7 . The method of  claim 1 , further comprising:
 determining an exploration reward based on: (i) a subsequent observation characterizing a state of the environment after the agent performs the selected action and (ii) one or more prior observations characterizing states of the environment prior to the agent performing the selected action; and   training the action selection neural network using a reinforcement learning technique based at least in part on the exploration reward.   
     
     
         8 . The method of  claim 7 , further comprising, prior to training the action selection neural network using the reinforcement learning technique based at least in part on the exploration reward:
 modifying the exploration reward based on the exploration importance factor.   
     
     
         9 . The method of  claim 8 , wherein modifying the exploration reward based on the exploration importance factor comprises:
 scaling the exploration reward by the exploration importance factor.   
     
     
         10 . The method of  claim 8 , further comprising:
 receiving a task reward that characterizes a progress of the agent towards accomplishing a task after the agent performs the selected action; and   determining an overall reward based on the task reward and the modified exploration reward;   wherein training the action selection neural network using the reinforcement learning technique based at least in part on the exploration reward comprises:
 training the action selection neural network using the reinforcement learning technique based on the overall reward. 
   
     
     
         11 . The method of  claim 10 , wherein determining the overall reward based on the task reward and the modified exploration reward comprises:
 determining the overall reward as a sum of the task reward and the modified exploration reward.   
     
     
         12 . The method of  claim 7 , wherein the reinforcement learning technique is a Retrace double Q-learning technique with prioritized experience replay. 
     
     
         13 . The method of  claim 1 , wherein the set of possible exploration importance factors is a discrete set. 
     
     
         14 . The method of  claim 7 , wherein determining the exploration reward comprises:
 providing the subsequent observation and the prior observations to an embedding neural network, wherein the embedding neural network is configured to process an input observation to generate an embedded representation of the input observation; and   determining the exploration reward based on, for each prior observation, a similarity measure between the embedded representation of the prior observation and the embedded representation of the subsequent observation.   
     
     
         15 . The method of  claim 14 , wherein the embedding neural network has been trained to process an input observation to generate an embedded representation of the input observation that characterizes aspects of the state characterized by the input observation that are controllable by the agent. 
     
     
         16 . The method of  claim 15 , wherein training the embedding neural network comprises jointly training the embedding neural network with an action prediction neural network, wherein:
 the action prediction neural network is configured to generate a prediction for an action that caused the state of the environment to transition from a state characterized by a given observation at a given time step to a state characterized by a subsequent observation at a subsequent time step by processing respective embedded representations of the given observation and the subsequent observation that are generated using the embedding neural network.   
     
     
         17 . The method of  claim 15 , wherein training the embedding neural network comprises jointly training the embedding neural network with a state prediction neural network, wherein:
 the state prediction neural network is configured to process an input comprising: (i) an embedded representation of a given observation that is generated using the embedding neural network and that characterizes the state of the environment at a given time step, and (ii) a representation of an action performed by the agent at the given time step, to generate an output characterizing a predicted subsequent state of the environment at a subsequent time step that follows the given time step.   
     
     
         18 . The method of  claim 14 , wherein determining the exploration reward based on, for each prior observation, a similarity measure between the embedded representation of the prior observation and the embedded representation of the subsequent observation comprises:
 determining a combined similarity measure based on the similarity measures between the embedded representation of the subsequent observation and the embedded representations of the prior observations; and   determining the exploration reward based on an inverse of the combined similarity measure.   
     
     
         19 . A system comprising:
 one or more computers; and   one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for selecting an action to be performed by an agent interacting with an environment, the operations comprising:   receiving an observation characterizing a current state of the environment;   processing: (i) the observation, and (ii) an exploration importance factor, using an action selection neural network to generate an action selection output, wherein:
 the action selection neural network implements a plurality of action selection policies indexed by a set of possible exploration importance factors; and 
 the plurality of action selection policies comprise an exploitative action selection policy and one or more exploratory action selection policies; and 
   selecting an action to be performed by the agent using the action selection output.   
     
     
         20 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting an action to be performed by an agent interacting with an environment, the operations comprising:
 receiving an observation characterizing a current state of the environment;   processing: (i) the observation, and (ii) an exploration importance factor, using an action selection neural network to generate an action selection output, wherein:
 the action selection neural network implements a plurality of action selection policies indexed by a set of possible exploration importance factors; and 
 the plurality of action selection policies comprise an exploitative action selection policy and one or more exploratory action selection policies; and 
   selecting an action to be performed by the agent using the action selection output.