Auto-tuning motion planning system for autonomous vehicles

Abstract

According to an embodiment, a system generates a number of sample trajectories from a trajectory sample space for a driving scenario. The system determines a reward based on a reward model for each of the sample trajectories, where the reward model is generated using a rank based conditional inverse reinforcement learning algorithm. The system ranks the sample trajectories based on the determined rewards. The system determines a highest ranked trajectory based on the ranking. The system selects the highest ranked trajectory to control the ADV autonomously according to the highest ranked trajectory.

What is claimed is: 
     
       1. A computer-implemented method to generate a motion planning cost function for an autonomous driving vehicle (ADV), the method comprising:
 collecting information for a driving environment surrounding the ADV using a plurality of sensors of the ADV; 
 generating a plurality of sample trajectories from a trajectory sample space for the driving environment; 
 determining a reward based on a reward model for each of the sample trajectories, wherein the reward model is generated using a rank based conditional inverse reinforcement learning algorithm, wherein the rank based conditional inverse reinforcement learning algorithm is a rank based inverse reinforcement learning conditional on a driving scenario such that the conditional inverse reinforcement learning algorithm is trainable scenario-wise, wherein the driving scenario includes a frame of a planning cycle: 
 wherein the reward model is also generated by: 
 generating a Siamese network for the reward model based on the plurality of sample trajectories in the trajectory sampling space and a target trajectory, wherein the target trajectory is an expert trajectory; and 
 applying an inverse reinforcement learning algorithm to the Siamese network to determine one or more weighting factors for the reward model to place the target trajectory in a highest ranking among the plurality of sample trajectories, wherein each of the weighting factors correspond to a respective feature for the reward model; 
 ranking the sample trajectories based on the determined rewards; 
 determining a highest ranked trajectory based on the ranking, from the sample trajectories based on the ranking; and 
 controlling the ADV autonomously according to the highest ranked trajectory. 
 
     
     
       2. The method of  claim 1 , wherein the reward model comprises a machine learning model comprises a multi-layer perceptron neural network model. 
     
     
       3. The method of  claim 2 , wherein the multi-layer perceptron neural network model includes an output layer to output a trajectory cost value. 
     
     
       4. The method of  claim 1 , wherein the reward model comprises a model based on a linear combination of features for the driving environment. 
     
     
       5. The method of  claim 4 , wherein the features comprise: acceleration, jerk, and velocity of the sample trajectory or a target trajectory, smoothness of roadway, or a distance from the sample trajectory or the target trajectory to surrounding obstacles observed on the roadway. 
     
     
       6. The method of  claim 1 , wherein the expert trajectory is generated based on a collection of human driven trajectories. 
     
     
       7. The method of  claim 1 , further comprising determining a plurality of features for each of the sample trajectories, and wherein the reward for each of the sample trajectories is determined based on the plurality of features. 
     
     
       8. The method of  claim 1 , wherein the plurality of sample trajectories is generated uniformly based on information for a driving environment of the ADV. 
     
     
       9. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
 collecting information for a driving environment surrounding the ADV using a plurality of sensors of the ADV; 
 generating a plurality of sample trajectories from a trajectory sample space for the driving environment; 
 determining a reward based on a reward model for each of the sample trajectories, wherein the reward model is generated using a rank based conditional inverse reinforcement learning algorithm, wherein the rank based conditional inverse reinforcement learning algorithm is a rank based inverse reinforcement learning conditional on a driving scenario such that the conditional inverse reinforcement learning algorithm is trainable scenario-wise, wherein the driving scenario includes a frame of a planning cycle; 
 wherein the reward model is also generated by: 
 generating a Siamese network for the reward model based on the plurality of sample trajectories in the trajectory sampling space and a target trajectory, wherein the target trajectory is an expert trajectory; and 
 applying an inverse reinforcement learning algorithm to the Siamese network to determine one or more weighting factors for the reward model to place the target trajectory in a highest ranking among the plurality of sample trajectories, wherein each of the weighting factors correspond to a respective feature for the reward model; 
 ranking the sample trajectories based on the determined rewards; 
 determining a highest ranked trajectory based on the ranking, from the sample trajectories based on the ranking; and 
 controlling the ADV autonomously according to the highest ranked trajectory. 
 
     
     
       10. The non-transitory machine-readable medium of  claim 9 , wherein the expert trajectory is generated based on collected human driven trajectories. 
     
     
       11. The non-transitory machine-readable medium of  claim 9 , wherein the operations further comprise: determining a plurality of features for each of the sample trajectories, and wherein the reward for each of the sample trajectories is determined based on the plurality of features. 
     
     
       12. The non-transitory machine-readable medium of  claim 9 , wherein the plurality of sample trajectories are generated uniformly based on information for a driving environment of the ADV. 
     
     
       13. A computer-implemented method to train a rewards model for an autonomous driving vehicle (ADV), the method comprising:
 determining a target trajectory based on driven trajectories collected from one or more vehicles; 
 generating a plurality of sample trajectories from a trajectory sample space for a driving environment of the target trajectory; and 
 generating a reward model by applying a rank based conditional inverse reinforcement learning algorithm to the sample trajectories and the target trajectory, wherein the reward model is used by an ADV to generate a driving trajectory to control the ADV, wherein the rank based conditional inverse reinforcement learning algorithm is a rank based inverse reinforcement learning conditional on a driving scenario such that the conditional inverse reinforcement learning algorithm is trainable scenario-wise, wherein the driving scenario includes a frame of a planning cycle; 
 wherein generating a reward model also includes applying a rank based conditional inverse reinforcement learning algorithm comprises: generating a Siamese network for the reward model based on the plurality of sample trajectories in the trajectory sampling space and a target trajectory; and 
 applying an inverse reinforcement learning algorithm to the Siamese network to determine one or more weighting factors for the reward model to place the target trajectory in a highest ranking among the plurality of sample trajectories, wherein each of the weighting factors correspond to a respective feature for the reward model, and
 ranking the sample trajectories based on the determined rewards; 
 determining a highest ranked trajectory based on the ranking, from the sample trajectories based on the ranking; and 
 controlling the ADV autonomously according to the highest ranked trajectory. 
 
 
     
     
       14. The method of  claim 13 , wherein the reward model comprises a machine learning model comprises a multi-layer perceptron neural network model. 
     
     
       15. The method of  claim 14 , wherein the multi-layer perceptron neural network model includes an output layer to output a trajectory cost value. 
     
     
       16. The method of  claim 13 , wherein the reward model comprises a model based on a linear combination of features for the driving environment. 
     
     
       17. The method of  claim 16 , wherein the features comprise: acceleration, jerk, and velocity of the sample trajectory or the target trajectory, smoothness of roadway, or a distance from the sample trajectory or the target trajectory to surrounding obstacles observed on the roadway. 
     
     
       18. The method of  claim 13 , wherein the expert trajectory is generated based on a collection of human driven trajectories. 
     
     
       19. The method of  claim 13 , further comprising determining a plurality of features for each of the sample trajectories, and wherein the reward model is generated based on the plurality of features. 
     
     
       20. The method of  claim 13 , wherein the plurality of sample trajectories are generated uniformly based on information for the driving environment of the target trajectory.