Differentiable and modular prediction and planning for autonomous machines

Abstract

In various examples, a motion planner include an analytical function to predict motion plans for a machine based on predicted trajectories of actors in an environment, where the predictions are differentiable with respect to parameters of a neural network of a motion predictor used to predict the trajectories. The analytical function may be used to determine candidate trajectories for the machine based on a predicted trajectory, to compute cost values for the candidate trajectories, and to select a reference trajectory from the candidate trajectories. For differentiability, a term of the analytical function may correspond to the predicted trajectory. A motion controller may use the reference trajectory to predict a control sequence for the machine using an analytical function trained to generate predictions that are differentiable with respect to at least one parameter of the analytical function used to compute the cost values.

What is claimed is: 
     
       1. A system comprising:
 one or more processing units to perform operations including:
 determining a future location of an agent in an environment using a neural network having parameters trained to predict the future location of the agent in the environment; 
 determining a plurality of candidate trajectories for a machine based at least on the future location of the agent in the environment; 
 computing cost values for the plurality of candidate trajectories using one or more analytical functions trained to generate predictions corresponding to the cost values using at least one input term derived from the future location, the cost values being differentiable with respect to the parameters of the neural network through the at least one input term; 
 selecting a trajectory from the plurality of candidate trajectories based at least on the cost values; and 
 
 performing one or more control operations for the machine using the trajectory. 
 
     
     
       2. The system of  claim 1 , wherein the operations further include:
 based at least on the cost values, computing, using the trajectory, a control sequence for the machine using one or more second analytical functions trained to generate second predictions corresponding to the control sequence, the second predictions being differentiable with respect to at least one parameter of the one or more analytical functions through at least one second input term of the one or more second analytical functions. 
 
     
     
       3. The system of  claim 2 , wherein the operations further include jointly training the one or more analytical functions, the one or more second analytical functions, and the neural network. 
     
     
       4. The system of  claim 1 , wherein the determining the plurality of candidate trajectories includes:
 sampling a current state of the machine in a state space and a set of terminal states for the machine to determine the plurality of candidate trajectories for the machine. 
 
     
     
       5. The system of  claim 1 , wherein the selecting the trajectory from the plurality of candidate trajectories includes classifying the trajectory as a target trajectory for the machine based at least on a categorical distribution corresponding to the plurality of candidate trajectories. 
     
     
       6. The system of  claim 1 , wherein the at least one input term computes a cost corresponding to the future location of the agent predicted using the neural network, the cost weighted using a corresponding weight value to compute a cost value of the cost values. 
     
     
       7. The system of  claim 1 , wherein the system is comprised in at least one of:
 a control system for an autonomous or semi-autonomous machine; 
 a perception system for an autonomous or semi-autonomous machine; 
 a system for performing simulation operations; 
 a system for performing digital twin operations; 
 a system for performing light transport simulation; 
 a system for performing collaborative content creation for 3D assets; 
 a system for performing deep learning operations; 
 a system implemented using an edge device; 
 a system implemented using a robot; 
 a system for performing conversational AI operations; 
 a system for generating synthetic data; 
 a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; 
 a system incorporating one or more virtual machines (VMs); 
 a system implemented at least partially in a data center; or 
 a system implemented at least partially using cloud computing resources. 
 
     
     
       8. The system of  claim 1 , the determining the future location includes applying, to the neural network, one or more initial locations of the agent, the one or more initial locations determined using sensor data generated using one or more sensors associated with the machine. 
     
     
       9. A method comprising:
 determining first predictions corresponding to one or more future states of one or more agents in an environment using at least one neural network having parameters trained to generate the first predictions; 
 computing one or more cost values for one or more candidate movements for a machine using one or more analytical functions having at least one input term derived from the one or more future states, the at least one input term including at least one parameter trained to generate second predictions corresponding to the one or more cost values, the one or more cost values being differentiable with respect to the parameters of the at least one neural network through that the at least one input term; and 
 performing one or more control operations for the machine based at least on the one or more cost values. 
 
     
     
       10. The method of  claim 9 , wherein the method further includes:
 based at least on the one or more cost values, computing, using the one or more candidate movements, a control sequence for the machine using one or more second analytical functions trained to generate third predictions corresponding to the control sequence, the third predictions being differentiable with respect to at least one parameter of the one or more analytical functions, wherein the one or more control operations correspond to the control sequence. 
 
     
     
       11. The method of  claim 9 , further comprising jointly training the one or more analytical functions and the at least one neural network based at least on backpropagating gradients through the at least one input term to the parameters of the at least one neural network. 
     
     
       12. The method of  claim 9 , wherein the one or more candidate movements are determined based at least on:
 sampling a current state of the machine in a state space and a set of terminal states for the machine to determine the one or more candidate movements for the machine. 
 
     
     
       13. The method of  claim 9 , wherein the performing the one or more control operations for the machine is based at least on:
 classifying a candidate movement of the one or more candidate movements as a target candidate movement for the machine based at least on a categorical distribution corresponding to the one or more candidate movements. 
 
     
     
       14. The method of  claim 9 , wherein the at least one input term computes a cost corresponding to the first predictions generated using the at least one neural network. 
     
     
       15. The method of  claim 9 , wherein the at least one input term includes a Gaussian radial basis function. 
     
     
       16. The method of  claim 9 , wherein the one or more cost values are based at least on one or more of:
 one or more predicted collisions between the machine and at least one agent of the one or more agents; 
 one or more distances between at least one state of the machine and at least one goal state for the machine; 
 one or more lateral lane deviation amounts; 
 one or more lane heading deviation amounts; or 
 one or more control effort amounts. 
 
     
     
       17. The method of  claim 9 , wherein the determining the first predictions includes applying, to the at least one neural network, one or more initial states of the one or more agents, the one or more initial states determined using sensor data generated using one or more sensors associated with the machine. 
     
     
       18. The method of  claim 17 , further including generating, using at least one second neural network, third predictions corresponding to the one or more initial states of the one or more agents in the environment. 
     
     
       19. A processor comprising:
 one or more circuits to perform one or more control operations for a machine based at least on one or more cost values determined for one or more candidate movements for the machine, 
 the one or more cost values determined using one or more analytical functions trained to generate first predictions corresponding to the one or more cost values using at least one input term derived from one or more future states of one or more agents, 
 the first predictions being differentiable with parameters of at least one neural network used to generate second predictions corresponding to the one or more future states through the at least one input term. 
 
     
     
       20. The processor of  claim 19 , wherein the one or more circuits are further to:
 based at least on the one or more cost values, compute, using the one or more candidate movements, a control sequence for the machine using one or more second analytical functions trained to generate third predictions corresponding to the control sequence, the third predictions being differentiable with respect to at least one parameter of the one or more analytical functions, wherein the one or more control operations correspond to the control sequence. 
 
     
     
       21. The processor of  claim 19 , wherein the one or more analytical functions and the at least one neural network are jointly trained. 
     
     
       22. The processor of  claim 19 , wherein the processor is comprised in at least one of:
 a control system for an autonomous or semi-autonomous machine; 
 a perception system for an autonomous or semi-autonomous machine; 
 a system for performing simulation operations; 
 a system for performing digital twin operations; 
 a system for performing light transport simulation; 
 a system for performing collaborative content creation for 3D assets; 
 a system for performing deep learning operations; 
 a system implemented using an edge device; 
 a system implemented using a robot; 
 a system for performing conversational AI operations; 
 a system for generating synthetic data; 
 a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; 
 a system incorporating one or more virtual machines (VMs); 
 a system implemented at least partially in a data center; or 
 a system implemented at least partially using cloud computing resources.