Learning motor primitives and training a machine learning system using a linear-feedback-stabilized policy

Abstract

A computer-implemented method of training a student machine learning system comprises receiving data indicating execution of an expert, determining one or more actions performed by the expert during the execution and a corresponding state-action Jacobian, and training the student machine learning system using a linear-feedback-stabilized policy. The linear-feedback-stabilized policy may be based on the state-action Jacobian. Also a neural network system for representing a space of probabilistic motor primitives, implemented by one or more computers. The neural network system comprises an encoder configured to generate latent variables based on a plurality of inputs, each input comprising a plurality of frames, and a decoder configured to generate an action based on one or more of the latent variables and a state.

What is claimed is: 
     
       1. A computer-implemented method of encoding behaviours for recall, the method comprising:
 obtaining a training trajectory representing an example behaviour, the training trajectory comprising, for each of a plurality of time steps during performance of the behaviour, (i) an observation representing a state of an environment at the time step and (ii) a training action performed at the time step; 
 for a particular time step t of the plurality of time steps: 
 generating action data a t  for the time step t from (i) an observation s t  representing the state of the environment at the time step t and one or more observations [s (t+1) , . . . , s t+k ] that represent states of the environment at time steps t+1, . . . t+k after the time step tin the training sequence, comprising:
 generating an encoder input x t  for the particular time step, wherein x t  comprises the observation at the time step t and the one or more observations representing states of the environment at a the one or more future time step from the training trajectory such that; 
 encoding the encoder input using an encoder neural network to determine parameters of a posterior distribution q t (z t |x t ) over a set of motor primitive latent variables; 
 sampling from the posterior distribution q t (z t |x t ) to determine a multi-dimensional motor primitive latent variable z t  for the particular time step; and 
 decoding (i) the multi-dimensional motor primitive latent variable z t  for the particular time step and (ii) the observation s t  at the particular time step using a generative neural network to generate action data a t  for the time step; and 
 training the encoder neural network and the generative neural network using an objective function dependent upon (i) the action data a t  output by the generative neural network for the particular time step and upon (ii) data representing the training action in the training trajectory at the particular time step. 
 
 
     
     
       2. The method as claimed in  claim 1  wherein the objective function further comprises a term dependent upon a difference between the posterior distribution and a prior distribution for the motor primitive latent variables. 
     
     
       3. The method as claimed in  claim 2  wherein the prior distribution comprises an autoregressive distribution such that at each time step the prior distribution depends on a combination of a times the prior distribution at a previous time step where |α|<1, and a noise component. 
     
     
       4. The method as claimed in  claim 1  further comprising recalling learned behaviour by encoding a sequence of observations of a target behaviour using the encoder neural network to generate a set of motor primitive latent variables for the target behaviour and then providing the set of motor primitive latent variables for the target behaviour to the generative neural network to provide action data for a sequence of actions to be implemented to perform to implement the target behaviour. 
     
     
       5. The method as claimed in  claim 1 , wherein the encoder input for the particular time step further comprises the multi-dimensional motor primitive latent variable for the time step preceding the particular time step in the training trajectory. 
     
     
       6. The method as claimed in  claim 1 , wherein the observations in the training trajectory are generated by applying first perturbations to observations in a nominal trajectory for the behaviour wherein the nominal trajectory is given by a sequence of nominal state action pairs {s* t , a* t } 1 . . . T  obtained by executing μ E (s) (the mean action of an expert in state s) recursively. 
     
     
       7. The method as claimed in  claim 6 , wherein the actions in the training trajectory are generated by applying second perturbations to actions in the nominal trajectory for the behaviour. 
     
     
       8. The method as claimed in  claim 7 , wherein the first perturbations are based on perturbations drawn from a perturbation distribution, and wherein the second perturbations are based on a state-action Jacobian of a policy used to generate the nominal trajectory and the perturbations drawn from the perturbation distribution. 
     
     
       9. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for encoding behaviours for recall, the operations comprising:
 obtaining a training trajectory representing an example behaviour, the training trajectory comprising, for each of a plurality of time steps during performance of the behaviour, (i) an observation representing a state of an environment at the time step and (ii) a training action performed at the time step; 
 for a particular time step t of the plurality of time steps: 
 generating action data a t  for the time step t from (i) an observation s t  representing the state of the environment at the time step t and one or more observations [s (t+1) , . . . , s t+k ] that represent states of the environment at time steps t+1, . . . t+k after the time step tin the training sequence, comprising:
 generating an encoder input x t  for the particular time step, wherein x t  comprises the observation at the time step t and the one or more observations representing states of the environment at a the one or more future time step from the training trajectory such that; 
 encoding the encoder input using an encoder neural network to determine parameters of a posterior distribution q t (z t |x t ) over a set of motor primitive latent variables; 
 sampling from the posterior distribution q t (z t |x t ) to determine a multi-dimensional motor primitive latent variable z t  for the particular time step; and 
 decoding (i) the multi-dimensional motor primitive latent variable z t  for the particular time step and (ii) the observation s t  at the particular time step using a generative neural network to generate action data a t  for the time step; and 
 training the encoder neural network and the generative neural network using an objective function dependent upon (i) the action data a t  output by the generative neural network for the particular time step and upon (ii) data representing the training action in the training trajectory at the particular time step. 
 
 
     
     
       10. The system as claimed in  claim 9  wherein the objective function further comprises a term dependent upon a difference between the posterior distribution and a prior distribution for the motor primitive latent variables. 
     
     
       11. The system as claimed in  claim 10  wherein the prior distribution comprises an autoregressive distribution such that at each time step the prior distribution depends on a combination of a times the prior distribution at a previous time step where |α|<1, and a noise component. 
     
     
       12. The system as claimed in  claim 9  further comprising recalling learned behaviour by encoding a sequence of observations of a target behaviour using the encoder neural network to generate a set of motor primitive latent variables for the target behaviour and then providing the set of motor primitive latent variables for the target behaviour to the generative neural network to provide action data for a sequence of actions to be implemented to perform to implement the target behaviour. 
     
     
       13. The system as claimed in  claim 9 , wherein the encoder input for the particular time step further comprises the multi-dimensional motor primitive latent variable for the time step preceding the particular time step in the training trajectory. 
     
     
       14. The system as claimed in  claim 9 , wherein the observations in the training trajectory are generated by applying first perturbations to observations in a nominal trajectory for the behaviour wherein the nominal trajectory is given by a sequence of nominal state action pairs {s* t , a* t } 1 . . . T  obtained by executing μ E (s) (the mean action of the expert in state s) recursively. 
     
     
       15. The system as claimed in  claim 14 , wherein the actions in the training trajectory are generated by applying second perturbations to actions in the nominal trajectory for the behaviour. 
     
     
       16. The system as claimed in  claim 15 , wherein the first perturbations are based on perturbations drawn from a perturbation distribution, and wherein the second perturbations are based on a state-action Jacobian of a policy used to generate the nominal trajectory and the perturbations drawn from the perturbation distribution. 
     
     
       17. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for encoding behaviours for recall, the operations comprising:
 obtaining a training trajectory representing an example behaviour, the training trajectory comprising, for each of a plurality of time steps during performance of the behaviour, (i) an observation representing a state of an environment at the time step and (ii) a training action performed at the time step;
 for a particular time step t of the plurality of time steps: 
 generating action data a t  for the time step t from (i) an observation s t  representing the state of the environment at the time step t and one or more observations [s (t+1) , . . . , s t+k ] that represent states of the environment at time steps t+1, . . . t+k after the time step tin the training sequence, comprising:
 generating an encoder input x t  for the particular time step, wherein x t  comprises the observation at the time step t and the one or more observations representing states of the environment at a the one or more future time step from the training trajectory such that; 
 encoding the encoder input using an encoder neural network to determine parameters of a posterior distribution q t (z t |x t ) over a set of motor primitive latent variables; 
 sampling from the posterior distribution q t (z t |x t ) to determine a multi-dimensional motor primitive latent variable z t  for the particular time step; and 
 decoding (i) the multi-dimensional motor primitive latent variable z t  for the particular time step and (ii) the observation s t  at the particular time step using a generative neural network to generate action data a t  for the time step; and 
 training the encoder neural network and the generative neural network using an objective function dependent upon (i) the action data a t  output by the generative neural network for the particular time step and upon (ii) data representing the training action in the training trajectory at the particular time step. 
 
 
 
     
     
       18. The one-or more non-transitory computer-readable storage media as claimed in  claim 17  wherein the objective function further comprises a term dependent upon a difference between the posterior distribution and a prior distribution for the motor primitive latent variables.