Multi-attention machine learning for object detection and classification

Abstract

Techniques for detecting, locating, and/or classifying objects based on multiple sensor data inputs received from different sensor modalities. The techniques may include receiving sensor data generated by different sensor modalities of a vehicle, the sensor data including at least first sensor data generated by a first sensor modality and second sensor data generated by a second sensor modality. In some examples, the sensor data may be input into a machine-learning pipeline. The machine-learning pipeline may be configured to determine locations of objects in an environment surrounding the vehicle based at least in part on a correlation, by the multi-attention component, of the first sensor data and the second sensor data. The techniques may also include receiving, from the machine-learning pipeline, an output indicating a location of an object in the environment.

What is claimed is: 
     
         1 . A system comprising:
 one or more processors; and   one or more non-transitory computer-readable media storing instructions that, when executed, cause the system to perform operations comprising:
 receiving first sensor data generated by a first modality of sensor of a vehicle; 
 receiving second sensor data generated by a second modality of sensor of the vehicle; 
 inputting the first sensor data and the second sensor data into a machine-learned model; 
 determining, by the machine-learned model, a first attention vector for the first sensor data; 
 determining, by the machine-learned model, a second attention vector for the second sensor data; and 
 determining, by the machine-learned model and based at least in part on the first attention vector and the second attention vector, at least one of a location or a classification of an object in an environment of the vehicle. 
   
     
     
         2 . The system of  claim 1 , further comprising:
 determining, by the machine-learned model, a first key vector, a first query vector, and a first value vector for the first sensor data;   determining, by the machine-learned model, a second key vector, a second query vector, and a second value vector for the second sensor data; and   determining portions of the first sensor data and the second sensor data;   wherein the first query vector represents weightings of portions of the first sensor data to determine self-attention for the first sensor data; and   wherein the second query vector represents weightings of portions of the second sensor data to determine self-attention for the second sensor data.   
     
     
         3 . The system of  claim 1 , wherein:
 the first modality is audio;   the second sensor data includes sensor data from at least two sensors, each sensor of the at least two sensors comprising a lidar sensor, a vision sensor, or a radar sensor; and   the operations further comprise fusing the two or more modalities of sensor data prior to the determining, by the machine-learned model, the second attention vector for the second sensor data.   
     
     
         4 . A method comprising:
 receiving first data generated by a first sensor modality;   receiving second data generated by a second sensor modality;   inputting the audio data and the spatial data into a machine-learned model;   determining, by at least a portion of the machine-learned model, a portion of the audio data and a portion of the spatial data;   associating, by the machine-learned model using an attention vector, the portion of the audio data with a portion of the spatial data;   determining, based at least in part on the portion of the audio data and the portion of the spatial data, at least one of a location or classification of an object in an environment of a vehicle.   
     
     
         5 . The method of  claim 4 , wherein the second sensor modality is a spatial sensor modality and the second data is collected by a plurality of sensors. 
     
     
         6 . The method of  claim 5 , wherein individual sensors of the plurality of sensors are of a same spatial sensor modality. 
     
     
         7 . The method of  claim 5 , wherein individual sensors of the plurality of sensors are of different spatial sensor modalities. 
     
     
         8 . The method of  claim 7 , wherein the different spatial sensor modalities comprise at least one of:
 vision;   lidar;   radar; or   infrared.   
     
     
         9 . The method of  claim 5 , wherein the second data comprises a combination of data collected from individual sensors of the plurality of sensors. 
     
     
         10 . The method of  claim 4 , further comprising:
 determining portions of the first data and the second data;   wherein a first query vector represents weightings of portions of the first data to determine self-attention for the first data; and   wherein a second query vector represents weightings of portions of the second data to determine self-attention for the second data.   
     
     
         11 . The method of  claim 4 , further comprising controlling operation of the vehicle based at least in part on the location of the object. 
     
     
         12 . The method of  claim 4 , further comprising generating, based at least in part on the portion of the second data and the portion of the first data, an intermediate value, wherein the at least one of the location or classification of the object is further based at least in part on the intermediate value. 
     
     
         13 . The method of  claim 12 , wherein the machine-learned model comprises a transformer and the intermediate value is generated based at least in part on a key vector or a value vector corresponding to the second data. 
     
     
         14 . The method of  claim 12 , wherein the intermediate value is a first intermediate value and the attention vector is a first attention vector associated with the portion of the spatial data, and the method further comprises:
 determining a second attention vector associated with the portion of the first data;   applying, by the machine-learned model, the second attention vector to the portion of the second data; and   generating, based at least in part on applying the second attention vector to the portion of the second data, a second intermediate value;   wherein the determining the location or classification of the object is based at least in part on the second intermediate value.   
     
     
         15 . The method of  claim 4 , wherein the second data comprises portions of second data that are at least one of tokenized features or down sampled data generated by the second sensor modality. 
     
     
         16 . The method of  claim 4 , wherein the first data is received from an array of multiple microphones of the vehicle. 
     
     
         17 . The method of  claim 4 , wherein the location of the object is a predicted location of the object, and wherein the machine-learned model is trained end-to-end based at least in part on:
 determining an actual location of the object;   determining that a difference between the actual location and the predicted location meets or exceeds a threshold difference; and   altering a parameter of the machine-learned model to reduce the difference below the threshold difference and obtain an end-to-end trained machine-learned model that is configured to determine the actual location of the object.   
     
     
         18 . One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations comprising:
 receiving audio data generated by a first modality of sensor of a vehicle;   receiving spatial data generated by a second modality of sensor of the vehicle;   inputting the audio data and the spatial data into a machine-learned model;   determining, by the machine-learned model, a first attention vector for the audio data;   determining, by the machine-learned model, a second attention vector for the spatial data; and   determining, by the machine-learned model and based at least in part on the first attention vector and the second attention vector, at least one of a location or a classification of an object in an environment of the vehicle.   
     
     
         19 . The one or more non-transitory computer-readable media of  claim 18 , wherein the machine-learned model further comprises a self-attention component and wherein data corresponding to portions of the audio data is generated by the self-attention component. 
     
     
         20 . The one or more non-transitory computer-readable media of  claim 18 , wherein the operations further comprise:
 determining, based at least in part on applying the second attention vector to the audio data, a first intermediate value;   determining, based at least in part on applying the first attention vector to the spatial data, a second intermediate value; and   wherein the determining the location or classification of the object is based at least in part on at least one of the first intermediate value or the second intermediate value.