Evaluating autonomous vehicle control system

Abstract

Techniques are disclosed for evaluating an autonomous vehicle (“AV”) control system by determining deviations between data generated using the AV control system and manual driving data. In many implementations, manual driving data captures action(s) of a vehicle controlled by a manual driver. Additionally or alternatively, multiple AV control systems can be evaluated by comparing deviations for each AV control system, where the deviations are determined using the same set of manual driving data.

What is claimed: 
     
       1. A method for evaluating an autonomous vehicle control system using previously captured manual driving data, the method implemented by one or more processors and comprising:
 capturing multiple instances of manual driving data captured from a vehicle which is manually driven in a conventional mode; 
 storing the multiple instances of manual driving data; 
 
       subsequent to the storing of the multiple instances of manual driving data,
 for each of a plurality of iterations:
 identifying a corresponding first instance of manual driving data captured at a first time step, the corresponding first instance of manual driving data being previously captured during control of a corresponding vehicle by a corresponding manual driver and comprising:
 corresponding current vehicle trajectory data that defines one or more aspects of a trajectory of the corresponding vehicle controlled by the corresponding manual driver at the first time step for the corresponding first instance, and 
 corresponding current environmental data that defines one or more aspects of an environment of the corresponding vehicle controlled by the corresponding manual driver at the first time step for the corresponding first instance; 
 
 processing the corresponding first instance of the manual driving data at the first time step, using the autonomous vehicle control system, to generate a corresponding predicted next instance of autonomous vehicle control system trajectory data defining one or more aspects of a predicted trajectory that would be implemented by the autonomous vehicle control system at a second time step right after the first time step; 
 comparing (a) the corresponding predicted next instance of autonomous vehicle control system trajectory data at the second time step to (b) a corresponding next instance of manual driver trajectory data, the corresponding next instance of manual driver trajectory data being previously captured at the second time step during the control of the corresponding vehicle by the corresponding manual driver, and following the corresponding first instance of manual driving data at the first time step; 
 determining a difference measure based on the comparing; and 
 evaluating the autonomous vehicle control system based on the difference measure from the plurality of iterations; 
 modifying at least one cost function of the autonomous vehicle control system based on the evaluation. 
 
 
     
     
       2. The method of  claim 1 , further comprising:
 determining, based on the evaluating of the autonomous vehicle control system, whether to deploy the autonomous vehicle control system in one or more autonomous vehicles. 
 
     
     
       3. The method of  claim 1 , further comprising:
 for each of a plurality of additional iterations:
 identifying the corresponding first instance of manual driving data; 
 processing the corresponding first instance of manual driving data, using an additional autonomous vehicle control system, to generate an additional corresponding predicted next instance of additional autonomous vehicle control system trajectory data defining one or more aspects of an additional trajectory that would be implemented by the additional autonomous vehicle control system in view of the first instance of manual driving data; 
 comparing (a) the additional corresponding predicted next instance of additional autonomous vehicle control system trajectory data to (b) the corresponding next instance of manual driver trajectory data, the corresponding next instance of manual driver trajectory data being previously captured during the control of the corresponding vehicle by the corresponding manual driver, and following the corresponding first instance of manual driving data; 
 determining an additional difference measure based on the comparing; and 
 evaluating the additional autonomous vehicle control system based on the additional difference measure from the additional plurality of iterations. 
 
 
     
     
       4. The method of  claim 1 , wherein the corresponding current environmental data that defines one or more aspects of the environment of the corresponding vehicle for the first instance includes sensor data captured using a sensor suite of the corresponding vehicle. 
     
     
       5. The method of  claim 1 , wherein the corresponding current vehicle trajectory data that defines the one or more aspects of the trajectory of the corresponding vehicle for the first instance includes one or more aspects of the trajectory of the corresponding vehicle for one or more previous instances. 
     
     
       6. The method of  claim 1 , wherein the corresponding current environmental data that defines the one or more aspects of the environment of the corresponding vehicle for the first instance includes one or more aspects of the environment of the corresponding vehicle for one or more previous instances. 
     
     
       7. The method of  claim 1 , further comprising:
 determining whether to classify the corresponding predicted next instance of autonomous vehicle control system trajectory data as a deviation based on the difference measure. 
 
     
     
       8. The method of  claim 7 , wherein the corresponding predicted next instance of autonomous vehicle control system trajectory data is a predicted next instance of autonomous vehicle control system trajectory based on a Gaussian distribution. 
     
     
       9. The method of  claim 8 , wherein determining whether to classify the corresponding predicted next instance of autonomous vehicle control system trajectory data as a deviation based on the difference measure comprises:
 determining a z-score value based on the corresponding predicted next instance of manual driver trajectory data and the predicted next instance of autonomous vehicle control system trajectory based on the Gaussian distribution; 
 determining the z-score value satisfies one or more conditions; and 
 in response determining the z-score value satisfies the one or more conditions, determining to classify the corresponding predicted next instance of autonomous vehicle control system trajectory Gaussian distribution as a deviation. 
 
     
     
       10. The method of  claim 8 , wherein determining whether to classify the corresponding predicted next instance of autonomous vehicle control system trajectory data as a deviation based on the difference measure comprises:
 determining a z-score value based on the corresponding predicted next instance of manual driver trajectory data and the predicted next instance of autonomous vehicle control system trajectory based on the Gaussian distribution; 
 determining the z-score value does not satisfy one or more conditions; and 
 in response to determining the z-score value does not satisfy the one or more conditions, determining to not classify the corresponding predicted next instance of autonomous vehicle control system trajectory data as a deviation. 
 
     
     
       11. The method of  claim 10 , further comprising:
 in response to determining to not classify the corresponding predicted next instance of autonomous vehicle control system trajectory data as a deviation; 
 comparing (a) the corresponding predicted next instance of autonomous vehicle control system trajectory data to (b) a corresponding further instance of manual driver trajectory data, the corresponding further instance of manual driver trajectory data being previously captured during the control of the corresponding vehicle by the corresponding manual driver, and following the corresponding next instance of manual driving data; and 
 determining an additional difference measure based on the comparing; and 
 determining whether to classify the corresponding predicted next instance of autonomous vehicle control system trajectory data as a deviation based on the additional difference measure. 
 
     
     
       12. The method of  claim 11 , further comprising:
 determining to classify the corresponding predicted next instance of autonomous vehicle control system trajectory data as a deviation based on the additional difference measure; and 
 in response to determining to classify the corresponding predicted next instance of autonomous vehicle control system trajectory data as a deviation based on the additional difference measure, determining a latency between the corresponding instance of manual driver trajectory data and the corresponding further instance of manual driver trajectory data. 
 
     
     
       13. The method of  claim 7 , further comprising:
 determining to classify the corresponding predicted next instance of autonomous vehicle control system trajectory data as a deviation based on the difference measure; 
 in response to determining to classify the corresponding predicted next instance of autonomous vehicle control system trajectory data as a deviation based on the difference measure, determining whether to classify the deviation as caused by the autonomous vehicle control system; and 
 wherein evaluating the autonomous vehicle control system based on the difference measure from the plurality of iterations comprises evaluating the autonomous vehicle control system based on the difference measure classified as caused by the autonomous vehicle control system from the plurality of iterations. 
 
     
     
       14. The method of  claim 8 , wherein determining to classify the corresponding predicted next instance of autonomous vehicle control system trajectory data as a deviation based on the difference measure comprises:
 determining a log likelihood value based on the corresponding predicted next instance of manual driver trajectory data and the predicted next instance of autonomous vehicle control system trajectory based on the Gaussian distribution; 
 determining the log likelihood value satisfies one or more conditions; and 
 in response to determining the log likelihood value satisfies the one or more conditions, determining to classify the corresponding predicted next instance of autonomous vehicle control system trajectory Gaussian distribution as a deviation. 
 
     
     
       15. A system including one or more processors that execute instructions, stored in an associated memory, the instructions when executed by the one or more processors evaluate an autonomous vehicle control system using previously captured manual driving data, comprising:
 capturing multiple instances of manual driving data captured from a vehicle which is manually driven; 
 storing the multiple instances of manual driving data; 
 subsequent to the storing of the multiple instances of manual driving data, for each of a plurality of iterations:
 identifying a corresponding first instance of manual driving data captured at a first time step, the corresponding first instance of manual driving data being previously captured during control of a corresponding vehicle by a corresponding manual driver and comprising:
 corresponding current vehicle trajectory data that defines one or more aspects of a trajectory of the corresponding vehicle controlled by the corresponding manual driver at the first time step for the corresponding first instance, and 
 corresponding current environmental data that defines one or more aspects of an environment of the corresponding vehicle controlled by the corresponding manual driver at the first time step for the corresponding first instance; 
 
 processing the corresponding first instance of the manual driving data at the first time step, using the autonomous vehicle control system, to generate a corresponding predicted next instance of autonomous vehicle control system trajectory data defining one or more aspects of a predicted trajectory that would be implemented by the autonomous vehicle control system at a second time step right after the first time step; 
 comparing (a) the corresponding predicted next instance of autonomous vehicle control system trajectory data at the second time step to (b) a corresponding next instance of manual driver trajectory data, the corresponding next instance of manual driver trajectory data being previously captured at the second time step during the control of the corresponding vehicle by the corresponding manual driver, and following the corresponding first instance of manual driving data at the first time step; 
 determining a difference measure based on the comparing; and 
 evaluating the autonomous vehicle control system based on the difference measure from the plurality of iterations; 
 modifying at least one cost function of the autonomous vehicle control system based on the evaluation. 
 
 
     
     
       16. The system of  claim 15 , further comprising:
 determining, based on the evaluating of the autonomous vehicle control system, whether to deploy the autonomous vehicle control system in one or more autonomous vehicles. 
 
     
     
       17. The system of  claim 15 , further comprising:
 for each of a plurality of additional iterations:
 identifying the corresponding first instance of manual driving data; 
 processing the corresponding first instance of manual driving data, using an additional autonomous vehicle control system, to generate an additional corresponding predicted next instance of additional autonomous vehicle control system trajectory data defining one or more aspects of an additional trajectory that would be implemented by the additional autonomous vehicle control system in view of the first instance of manual driving data; 
 comparing (a) the additional corresponding predicted next instance of additional autonomous vehicle control system trajectory data to (b) the corresponding next instance of manual driver trajectory data, the corresponding next instance of manual driver trajectory data being previously captured during the control of the corresponding vehicle by the corresponding manual driver, and following the corresponding first instance of manual driving data; 
 determining an additional difference measure based on the comparing; and 
 evaluating the additional autonomous vehicle control system based on the additional difference measure from the additional plurality of iterations. 
 
 
     
     
       18. The system of  claim 15 , wherein the corresponding current environmental data that defines one or more aspects of the environment of the corresponding vehicle for the first instance includes sensor data captured using a sensor suite of the corresponding vehicle. 
     
     
       19. A non-transitory computer-readable storage medium comprising instructions executable by one or more processors of a computing system to evaluate an autonomous vehicle control system using previously captured manual driving data, by:
 capturing multiple instances of manual driving data captured from a vehicle which is manually driven; 
 storing the multiple instances of manual driving data; 
 for each of a plurality of iterations:
 identifying a corresponding first instance of manual driving data captured at a first time step, the corresponding first instance of manual driving data being previously captured during control of a corresponding vehicle by a corresponding manual driver and comprising:
 corresponding current vehicle trajectory data that defines one or more aspects of a trajectory of the corresponding vehicle controlled by the corresponding manual driver at the first time step for the corresponding first instance, and 
 corresponding current environmental data that defines one or more aspects of an environment of the corresponding vehicle controlled by the corresponding manual driver at the first time step for the corresponding first instance; 
 
 processing the corresponding first instance of the manual driving data at the first time step, using the autonomous vehicle control system, to generate a corresponding predicted next instance of autonomous vehicle control system trajectory data defining one or more aspects of a predicted trajectory that would be implemented by the autonomous vehicle control system at a second time step right after the first time step; 
 comparing (a) the corresponding predicted next instance of autonomous vehicle control system trajectory data at the second time step to (b) a corresponding next instance of manual driver trajectory data, the corresponding next instance of manual driver trajectory data being previously captured at the second time step during the control of the corresponding vehicle by the corresponding manual driver, and following the corresponding first instance of manual driving data at the first time step; 
 determining a difference measure based on the comparing; and 
 evaluating the autonomous vehicle control system based on the difference measure from the plurality of iterations; 
 modifying at least one cost function of the autonomous vehicle control system based on the evaluation.