Systems and methods for interaction-based trajectory prediction

Abstract

Systems and methods for predicting interactions between objects and predicting a trajectory of an object are presented herein. A system can obtain object data associated with a first object and a second object. The object data can have position data and velocity data for the first object and the second object. Additionally, the system can process the obtained object data to generate a hybrid graph using a graph generator. The hybrid graph can have a first node indicative of the first object and a second node indicative of the second object. Moreover, the system can process, using an interaction prediction model, the generated hybrid graph to predict an interaction type between the first node and the second node. Furthermore, the system can process, using a graph neural network model, the predicted interaction type between the first node and the second node to predict a trajectory of the first object.

What is claimed is: 
     
       1. A computing system for an autonomous vehicle, the computing system comprising:
 one or more processors; and
 one or more non-transitory computer-readable medium storing instructions for execution by the one or more processors to cause the computing system to perform operations, the operations comprising:
 obtaining object data associated with a first object and a second object, the object data having position data and velocity data for the first object and the second object; 
 processing the obtained object data to generate a hybrid graph, the hybrid graph having a first node indicative of the first object and a second node indicative of the second object; 
 processing, using an interaction prediction model, the generated hybrid graph to predict an interaction type for edges in the hybrid graph including an edge between the first node and the second node, wherein the interaction type is predicted from a predetermined set of discrete interaction types; 
 processing, using a graph neural network model, the predicted interaction type between the first node and the second node to predict a trajectory of the first object; and 
 controlling motion of the autonomous vehicle based on a motion plan determined from the trajectory of the first object. 
 
 
 
     
     
       2. The computing system of  claim 1 , the operations further comprising:
 determining a loss function based on the interaction type between the first node and the second node; and 
 modifying one or more of parameter values of the interaction prediction model based at least in part on the loss function. 
 
     
     
       3. The computing system of  claim 1 , the operations comprising:
 obtaining traffic element data associated with a traffic element, the traffic element data having map data of an area surrounding the traffic element; 
 processing, using a graph generator, the obtained traffic element data to generate the hybrid graph, the hybrid graph having a third node indicative of the traffic element; and 
 processing, using the interaction prediction model, the generated hybrid graph to predict an interaction type between the first node and the third node; and 
 processing, using the graph neural network model, the predicted interaction type between the first node and the third node to predict a trajectory of the first object. 
 
     
     
       4. The computing system of  claim 3 , the operations further comprising:
 processing, using the interaction prediction model, the generated hybrid graph to predict an interaction type between the first node and the third node when the first object is less than a predefined distance from the traffic element, and an interaction type between the second node and third node when the second object is less than the predefined distance from the traffic element; 
 determining a cross-entropy loss function based at least in part on the interaction type between the first node and the second node, the interaction type between the first node and third node, or the interaction type between the second node and third node; and 
 modifying one or more of parameter values of the interaction prediction model based at least in part on the cross-entropy loss function. 
 
     
     
       5. The computing system of  claim 3 , the operations further comprising:
 processing, using the interaction prediction model, the generated hybrid graph to predict an interaction type between the first node and the third node, and an interaction type between the second node and the third node; 
 processing, using the graph neural network model, the interaction type between the first node and the second node, the interaction type between the first node and third node, or the interaction type between the second node and third node to predict a trajectory of the first node and a trajectory of the second node; 
 determining a Huber loss function based on the predicted trajectory of the first node or the predicted trajectory of the second node; and 
 modifying one or more of parameter values of the graph neural network model based at least in part on the Huber loss function. 
 
     
     
       6. The computing system of  claim 3 , wherein the obtained traffic element data indicates that a traffic light state of the traffic element is an unknown traffic light, the operations further comprising:
 determining, using a light prediction model, that the traffic light state of the traffic element is either a green traffic light, a yellow traffic light, or a red traffic light; 
 updating, using the graph generator, the generated hybrid graph based on the determined traffic light state; 
 updating, using the interaction prediction model, the interaction type between the first object and the second object based on the updated hybrid graph; and 
 updating, using the graph neural network model, the predicted trajectory of the first object based on the updated interaction type between the first object and the second object. 
 
     
     
       7. The computing system of  claim 3 , wherein the map data includes lane boundary data, left turn region data, right turn region data, motion path data, drivable area data, intersection data. 
     
     
       8. The computing system of  claim 3 , wherein the traffic element is a red traffic light, a yellow traffic light, a green traffic light, an unknown traffic light, a stop sign, or a yield sign. 
     
     
       9. The computing system of  claim 3 , wherein the interaction type between the first node and the second node is (i) for the first object to ignore when the predicted trajectory of the first object and the predicted trajectory of the second object do not intersect, (ii) for the first object to go when the predicted trajectory of the first object and the predicted trajectory of the second object do intersect and the first object arrives at the traffic element before the second object, or (iii) for the first object to yield when the predicted trajectory of the first object and the predicted trajectory of the second object do intersect and the first object arrives at the traffic element after the second object. 
     
     
       10. The computing system of  claim 1 , wherein the operations further comprising implementing a motion plan for the second object based on the predicted trajectory of the first object. 
     
     
       11. The computing system of  claim 1 , wherein the first object and the second object comprise a first vehicle and a second vehicle. 
     
     
       12. The computing system of  claim 11 , wherein the first vehicle and the second vehicle are both moving vehicles, and parked vehicles are not included as nodes in the generated hybrid graph. 
     
     
       13. The computing system of  claim 1 , wherein the hybrid graph includes a directional edge from the first node and the second node, the directional edge having a relative position and velocity of the first object in relation to the second object. 
     
     
       14. The computing system of  claim 1 , wherein the hybrid graph includes past trajectory data of the first node associated with the past trajectory of the first object. 
     
     
       15. A computer-implemented method for controlling motion of an autonomous vehicle comprising:
 obtaining object data associated with a first object and a second object, the object data having position data and velocity data for the first object and the second object; 
 processing, using a graph generator, the obtained object data to generate a hybrid graph, the hybrid graph having a first node indicative of the first object and a second node indicative of the second object;
 processing, using an interaction prediction model, the generated hybrid graph to predict an interaction type for edges in the hybrid graph including an edge between the first node and the second node, wherein the interaction type is predicted from a predetermined set of discrete interaction types; 
 
 processing, using a graph neural network model, the predicted interaction type between the first node and the second node to predict a trajectory of the first object; and 
 controlling motion of the autonomous vehicle based on a motion plan determined from the trajectory of the first object. 
 
     
     
       16. The computer-implemented method of  claim 15 , the method further comprising:
 obtaining traffic element data associated with a traffic element, the traffic element data having map data of an area surrounding the traffic element; 
 processing, using the graph generator, the obtained traffic element data to generate the hybrid graph, the hybrid graph having a third node indicative of the traffic element; and
 processing, using the interaction prediction model, the generated hybrid graph to predict an interaction type between the first node and the third node; and 
 processing, using the graph neural network model, the predicted interaction type between the first node and the third node to predict a trajectory of the first object. 
 
 
     
     
       17. The computer-implemented method of  claim 16 , the method further comprising:
 processing, using the interaction prediction model, the generated hybrid graph to predict an interaction type between the first node and the third node when the first object is less than a predefined distance from the traffic element, and an interaction type between the second node and third node when the second object is less than the predefined distance from the traffic element; 
 determining a cross-entropy loss function based at least in part on the interaction type between the first node and the second node, the interaction type between the first node and third node, or the interaction type between the second node and third node; and 
 modifying one or more of parameter values of the interaction prediction model based at least in part on the cross-entropy loss function. 
 
     
     
       18. The computer-implemented method of  claim 16 , the method further comprising:
 processing, using the interaction prediction model, the generated hybrid graph to predict an interaction type between the first node and the third node, and an interaction type between the second node and the third node; 
 processing, using the graph neural network model, the interaction type between the first node and the second node, the interaction type between the first node and third node, or the interaction type between the second node and third node to predict a trajectory of the first node and a trajectory of the second node; 
 determining a Huber loss function based on the predicted trajectory of the first node and the predicted trajectory of the second node; and 
 modifying one or more of parameter values of the graph neural network model based at least in part on the Huber loss function. 
 
     
     
       19. The computer-implemented method of  claim 16 , wherein the obtained traffic element data indicates that a traffic light state of the traffic element is an unknown traffic light, and wherein the method further comprises:
 determining, using a light prediction model, that the traffic light state of the traffic element is either a green traffic light, a yellow traffic light, or a red traffic light; 
 updating, using the graph generator, the generated hybrid graph based on the determined traffic light state; 
 updating, using the interaction prediction model, the interaction type between the first object and the second object based on the updated hybrid graph; and 
 updating, using the graph neural network model, the predicted trajectory of the first object based on the updated interaction type between the first object and the second object. 
 
     
     
       20. An autonomous vehicle comprising:
 one or more processors; and 
 one or more non-transitory computer-readable medium storing instructions for execution by the one or more processors to cause the one or more processors to perform operations, the operations comprising:
 obtaining object data associated with a first object and a second object, the object data having position data and velocity data for the first object and the second object; 
 obtaining traffic element data associated with a traffic element, the traffic element data having map data of an area surrounding the traffic element; 
 processing the obtained object data and the obtained traffic element data to generate a hybrid graph, the hybrid graph having a first node indicative of the first object, a second node indicative of the second object, and a third node indicative of the traffic element; 
 
 processing, using a prediction model, the generated hybrid graph to predict an interaction type for edges in the hybrid graph including an edge between the first node and the second node and an edge between the first node and the third node, wherein the interaction type is predicted from a predetermined set of discrete interaction types;
 processing, using the prediction model, the predicted interaction type between the first node and the second node, and the predicted interaction type between the first node and the third node to predict a trajectory of the first object; and 
 controlling motion of the autonomous vehicle based on a motion plan determined from the trajectory of the first object.