Systems for testing and training autonomous vehicles

Abstract

A computer implemented method of path verification in a computer system is described. A user provides input to mark a displayed image of a scenario. A path is generated representing the trajectory of a vehicle. Control points along the path are recorded, each control point associated with a vehicle position and target speed. A vehicle position end target speed of two control points is used to calculate at least one path verification parameter which defines how a vehicle travelling along the path would behave. The at least one verification parameter is compared with a corresponding threshold value; and an alert is generated to the user at the user interface when the at least one path verification parameter exceeds the corresponding threshold value.

The invention claimed is: 
     
       1. A computer implemented method of path verification in a computer system having a user interface configured to display an image to a user and to detect user input for marking the image to generate a path, the method comprising:
 receiving user input at the user interface to mark a displayed image of a scenario configured to test behavior of a vehicle in a simulated environment; 
 generating a path defined by the user input and representing a trajectory of the vehicle in the scenario; 
 recording in a computer memory control points along the path, each control point associated with a vehicle position and target speed defined by the user input; 
 in a processor, calculating from the vehicle position and target speed of two control points a first value of at least one path verification parameter which defines how a vehicle travelling along the path would behave; 
 comparing the first value of the at least one path verification parameter with a corresponding threshold value; and, based on the first value exceeding the corresponding threshold value, carrying out at least one of: 
 generating an alert to the user at the user interface; or 
 automatically modifying the path so that the threshold value is not exceeded by a second value of the at least one path verification parameter associated with the modified path; and 
 testing the behavior of the vehicle, wherein the testing includes simulating the behavior of the vehicle along the generated path representing the trajectory of the vehicle in the scenario. 
 
     
     
       2. The method of  claim 1  wherein the step of receiving user input comprises receiving at an editing interface of the computer system user input for marking multiple locations to create the path, and the step of generating the path comprises generating at least one path which passes through the multiple locations. 
     
     
       3. The method of  claim 2  comprising rendering the at least one path in the image which is displayed at the user interface. 
     
     
       4. The method of  claim 1 , comprising receiving user input defining adaption data for path verification, the adaption data selected from vehicle type specific data and driving comfort related data and storing the adaption data in computer storage. 
     
     
       5. The method of  claim 4  wherein the vehicle type specific data comprises one of vehicle wheel-base and maximum steering angle. 
     
     
       6. The method of  claim 4  wherein the driving comfort related data comprises at least one of maximum turn rate, maximum lateral acceleration and maximum longitudinal acceleration. 
     
     
       7. The method of  claim 1 , further comprising, generating and storing in computer storage at least one observation for at least one location on the path when the first value of the at least one path verification parameter exceeds the corresponding threshold value. 
     
     
       8. The method of  claim 1 , wherein the step of calculating the first value of the at least one path verification parameter comprises using four consecutive control points to produce a spline and verifying a segment between two mid-points of each spline on the path. 
     
     
       9. The method  claim 1 , wherein the at least one path verification parameter is selected from: longitudinal acceleration, curvature, steering angle, lateral acceleration, and turn rate. 
     
     
       10. The method of  claim 1 , wherein the two control points are adjacent control points. 
     
     
       11. The method of  claim 1  wherein the step of receiving user input comprises receiving annotation input from the user at the user interface for annotating a scene captured by the vehicle in a real-world environment, and the step of generating the path comprises generating a path from scene data. 
     
     
       12. The method of  claim 11  wherein the user is annotating motion of vehicles in a video captured by the vehicle in the real world environment and rendered on the user interface to mark the path on the user interface. 
     
     
       13. The method of  claim 11  wherein the step of generating the path comprises generating the path from time sequenced scenes captured by the vehicle in the real world environment. 
     
     
       14. A computer system comprising:
 one or more processors; and 
 computer memory comprising instructions executable by the one or more processors to implement: 
 a user interface configured to receive user input to mark a displayed image of a scenario configured to test behavior of a vehicle in a simulated environment; 
 wherein the computer memory records control points along the path, each control point associated with a vehicle position defined by the user input and target speed defined by the user input; and 
 wherein the one or more processors is configured to carry out the operations of: 
 generating a path defined by the user input and representing a trajectory of the vehicle in the scenario; 
 calculating from the vehicle position and target speed of two control points a first value of at least one path verification parameter which defines how a vehicle travelling along the path would behave; and 
 comparing the first value of the at least one path verification parameter with a corresponding threshold value; and, based on the first value exceeding the corresponding threshold value, carrying out at least on of: 
 generating an alert to the user at the user interface, or automatically modifying the path so that the threshold value is not exceeded by a second value of the at least one path verification parameter associated with the modified path; and 
 testing the behavior of the vehicle, wherein the testing includes simulating the behavior of the vehicle along the generated path representing the trajectory of the vehicle in the scenario. 
 
     
     
       15. The system of  claim 14 , wherein the instructions are further executable by the one or more processors to implement an editing interface configured to receive user input for marking multiple locations to create the path, and wherein the step of generating the path comprises generating at least one path which passes through the multiple locations. 
     
     
       16. The system of  claim 15 , wherein the instructions are further executable by the one or more processors to render the at least one path in the image which is displayed at the user interface. 
     
     
       17. The system of  claim 14 , wherein the memory comprises storage for holding adaption data for path verification, the adaption data selected from vehicle type specific data and driving comfort related data and storing the adaption data in the memory. 
     
     
       18. The system of  claim 17 , wherein the driving comfort related data comprises at least one of maximum turn rate, maximum lateral acceleration and maximum longitudinal acceleration. 
     
     
       19. The system of  claim 14 , wherein the memory holds at least one observation for at least one location on the path when the first value of the at least one path verification parameter exceeds the corresponding threshold value. 
     
     
       20. A computer program product comprising computer executable code stored on a non-transitory computer-readable medium which, when executed by a computer carries out steps of:
 receiving user input at a user interface to mark a displayed image of a scenario configured to test behavior of a vehicle in a simulated environment; 
 generating a path defined by the user input and representing a trajectory of the vehicle in the scenario; 
 recording in a computer memory control points along the path, each control point associated with a vehicle position defined by the user input and target speed defined by the user input; 
 in a processor, calculating from the vehicle position and target speed of two control points a first value of at least one path verification parameter which defines how a vehicle travelling along the path would behave; 
 comparing the first value of the at least one path verification parameter with a corresponding threshold value; and, based on the first value exceeding the corresponding threshold value, carrying out at least one of: 
 generating an alert to the user at the user interface; or 
 automatically modifying the path so that the threshold value is not exceeded by a second value of the at least one path verification parameter associated with the modified path; and 
 testing the behavior of the vehicle, wherein the testing includes simulating the behavior of the vehicle along the generated path representing the trajectory of the vehicle in the scenario.