System and method for evaluating motion prediction models

Abstract

A computing system can receive motion prediction data from a vehicle, where the motion prediction data is generated by a motion prediction model executing on the vehicle. Based on the motion prediction data, the system can determine predicted trajectories for a plurality of entities in a surrounding environment of the vehicle. The system can evaluate a prediction performance of the motion prediction model by (i) matching, for each respective entity of the plurality of entities, a predicted endpoint of each predicted trajectory of the set of predicted trajectories to one or more underlying lanes of the road segment, (ii) matching a ground truth future position of the entity to one or more underlying lanes of the underlying lane topology, and (iii) determining a distance along one or more lane segments between the lane(s) matched to the predicted endpoint and the lane(s) matched to the ground truth future position.

What is claimed is: 
     
         1 . A computing system comprising:
 one or more processors;   a memory storing instructions that, when executed by the one or more processors, cause the computing system to:   receive motion prediction data from a vehicle, the motion prediction data being generated by a motion prediction model executing on the vehicle as the vehicle operates on a road network;   based on the motion prediction data, determine a set of predicted trajectories for each of a plurality of entities in a surrounding environment of the vehicle;   determine an underlying lane topology of a road segment on which the vehicle operates;   evaluate a prediction performance of the motion prediction model by   (i) matching, for each respective entity of the plurality of entities, a predicted endpoint of each predicted trajectory of the set of predicted trajectories to one or more underlying lanes of the underlying lane topology of the road segment, (ii) matching a ground truth future position of the respective entity to one or more underlying lanes of the underlying lane topology, and (iii) determining a distance along one or more lane segments between the one or more underlying lanes matched to the predicted endpoint and the one or more underlying lanes matched to the ground truth future position;   wherein the executed instructions cause the computing system to match the predicted endpoint of the set of predicted trajectories to the one or more underlying lanes by calculating an assignment confidence for each of the one or more underlying lanes based on the set of predicted trajectories by dynamically calculating (i) a distance-based assignment confidence, and (ii) an orientation-based assignment confidence for each surrounding lane segment of the predicted endpoint;   wherein the one or more underlying lanes correspond to highest assignment confidences based on dynamically calculating the distance-based assignment confidence and the orientation-based assignment confidence;   wherein execution of the motion prediction model causes the vehicle to dynamically generate a motion plan based, at least in part, on the set of predicted trajectories for each of the plurality of entities in the surrounding environment of the vehicle; and   autonomously operate a set of control mechanisms of the vehicle to execute the motion plan.   
     
     
         2 . The computing system of  claim 1 , wherein the executed instructions further cause the computing system to:
 receive recorded sensor data from the vehicle, the recorded sensor data indicating a ground truth trajectory for each of the plurality of entities; and   based on the ground truth trajectory for each of the plurality of entities, determine the ground truth future position of each of the plurality of entities.   
     
     
         3 . The computing system of  claim 1 , wherein the set of control mechanisms comprises a plurality of the following: a braking system of the vehicle, a steering system of the vehicle, an acceleration system of the vehicle, or a signaling system of the vehicle. 
     
     
         4 . The computing system of  claim 1 , wherein the executed instructions cause the computing system to evaluate the prediction performance of each of a plurality of motion prediction models to determine a most accurate motion prediction model for execution on autonomous vehicles. 
     
     
         5 . The computing system of  claim 1 , wherein the plurality of entities comprise other vehicles operating along the road segment. 
     
     
         6 . A non-transitory computer readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to:
 receive motion prediction data from a vehicle, the motion prediction data being generated by a motion prediction model executing on the vehicle as the vehicle operates on a road network;   based on the motion prediction data, determine a set of predicted trajectories for each of a plurality of entities in a surrounding environment of the vehicle;   determine an underlying lane topology of a road segment on which the vehicle operates;   evaluate a prediction performance of the motion prediction model by (i) matching, for each respective entity of the plurality of entities, a predicted endpoint of each predicted trajectory of the set of predicted trajectories to one or more underlying lanes of the underlying lane topology of the road segment, (ii) matching a ground truth future position of the respective entity to one or more underlying lanes of the underlying lane topology, and (iii) determining a distance along one or more lane segments between the one or more underlying lanes matched to the predicted endpoint and the one or more underlying lanes matched to the ground truth future position;   wherein the executed instructions cause the computing system to match the predicted endpoint of the set of predicted trajectories to the one or more underlying lanes by calculating an assignment confidence for each of the one or more underlying lanes based on the set of predicted trajectories by dynamically calculating (i) a distance-based assignment confidence, and (ii) an orientation-based assignment confidence for each surrounding lane segment of the predicted endpoint;   wherein the one or more underlying lanes correspond to highest assignment confidences based on dynamically calculating the distance-based assignment confidence and the orientation-based assignment confidence;   wherein execution of the motion prediction model causes the vehicle to dynamically generate a motion plan based, at least in part, on the set of predicted trajectories for each of the plurality of entities in the surrounding environment of the vehicle; and   autonomously operate a set of control mechanisms of the vehicle to execute the motion plan.   
     
     
         7 . The non-transitory computer readable medium of  claim 6 , wherein the executed instructions further cause the computing system to:
 receive recorded sensor data from the vehicle, the recorded sensor data indicating a ground truth trajectory for each of the plurality of entities; and   based on the ground truth trajectory for each of the plurality of entities, determine the ground truth future position of each of the plurality of entities.   
     
     
         8 . The non-transitory computer readable medium of  claim 6 , wherein the set of control mechanisms comprises a plurality of the following: a braking system of the vehicle, a steering system of the vehicle, an acceleration system of the vehicle, or a signaling system of the vehicle. 
     
     
         9 . The non-transitory computer readable medium of  claim 6 , wherein the executed instructions cause the computing system to evaluate the prediction performance of each of a plurality of motion prediction models to determine a most accurate motion prediction model for execution on autonomous vehicles. 
     
     
         10 . The non-transitory computer readable medium of  claim 6 , wherein the plurality of entities comprise other vehicles operating along the road segment. 
     
     
         11 . A computer-implemented method, comprising:
 receiving motion prediction data from a vehicle, the motion prediction data being generated by a motion prediction model executing on the vehicle as the vehicle operates on a road network;   based on the motion prediction data, determining a set of predicted trajectories for each of a plurality of entities in a surrounding environment of the vehicle;   determining an underlying lane topology of a road segment on which the vehicle operates;   evaluating a prediction performance of the motion prediction model by (i) matching, for each respective entity of the plurality of entities, a predicted endpoint of each predicted trajectory of the set of predicted trajectories to one or more underlying lanes of the underlying lane topology of the road segment, (ii) matching a ground truth future position of the respective entity to one or more underlying lanes of the underlying lane topology, and (iii) determining a distance along one or more lane segments between the one or more underlying lanes matched to the predicted endpoint and the one or more underlying lanes matched to the ground truth future position;   wherein the executed instructions cause the computing system to match the predicted endpoint of the set of predicted trajectories to the one or more underlying lanes by calculating an assignment confidence for each of the one or more underlying lanes based on the set of predicted trajectories by dynamically calculating (i) a distance-based assignment confidence, and (ii) an orientation-based assignment confidence for each surrounding lane segment of the predicted endpoint;   wherein the one or more underlying lanes correspond to highest assignment confidences based on dynamically calculating the distance-based assignment confidence and the orientation-based assignment confidence;   wherein execution of the motion prediction model causes the vehicle to dynamically generate a motion plan based, at least in part, on the set of predicted trajectories for each of the plurality of entities in the surrounding environment of the vehicle; and   autonomously operate a set of control mechanisms of the vehicle to execute the motion plan.   
     
     
         12 . The method of  claim 11 , further comprising:
 receiving recorded sensor data from the vehicle, the recorded sensor data indicating a ground truth trajectory for each of the plurality of entities; and   based on the ground truth trajectory for each of the plurality of entities, determining the ground truth future position of each of the plurality of entities.