Systems and methods for sensor data processing and object detection and motion prediction for robotic platforms

Abstract

Systems and methods are disclosed for detecting and predicting the motion of objects within the surrounding environment of a system such as an autonomous vehicle. For example, an autonomous vehicle can obtain sensor data from a plurality of sensors comprising at least two different sensor modalities (e.g., RADAR, LIDAR, camera) and fused together to create a fused sensor sample. The fused sensor sample can then be provided as input to a machine learning model (e.g., a machine learning model for object detection and/or motion prediction). The machine learning model can have been trained by independently applying sensor dropout to the at least two different sensor modalities. Outputs received from the machine learning model in response to receipt of the fused sensor samples are characterized by improved generalization performance over multiple sensor modalities, thus yielding improved performance in detecting objects and predicting their future locations, as well as improved navigation performance.

What is claimed is: 
     
       1. A computer-implemented method for training an object detection model, comprising:
 (a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities; 
 (b) independently applying sensor dropout to the at least two different sensor modalities of the sensor data, including modifying intensity values of at least one of the at least two different sensor modalities; 
 (c) fusing the sensor data from the at least two different sensor modalities with sensor dropout independently applied thereto to generate a fused sensor sample; 
 (d) processing the fused sensor sample with at least a portion of the object detection model; and 
 (e) updating one or more weights of the object detection model based on labels associated with the sensor data; and 
 (f) employing the object detection model by a robotic platform operating within an environment. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein (b) comprises independently applying sensor dropout to each of the at least two different sensor modalities at a fixed probability associated with the sensor modality. 
     
     
       3. The computer-implemented method of  claim 1 , wherein the plurality of sensors comprise a RADAR system, a LIDAR system, and a camera. 
     
     
       4. The computer-implemented method of  claim 3 , wherein:
 the at least two different sensor modalities comprise at least one of the RADAR system or the camera; and 
 (b) comprises zeroing out a final feature vector for a portion of the sensor data obtained from the at least one of the RADAR system or the camera. 
 
     
     
       5. The computer-implemented method of  claim 3 , wherein:
 the at least two different sensor modalities comprise the LIDAR system; and 
 (b) comprises replacing a LIDAR intensity value with a sentinel value for a portion of the sensor data obtained from the LIDAR system. 
 
     
     
       6. The computer-implemented method of  claim 2 , wherein the fixed probability associated with the sensor modality for at least one of the two different sensor modalities is zero. 
     
     
       7. The computer-implemented method of  claim 1 , wherein the object detection model comprises an end-to-end model that is configured to jointly perform object detection and motion prediction. 
     
     
       8. The computer-implemented method of  claim 1 , wherein the robotic platform comprises an autonomous vehicle. 
     
     
       9. The computer-implemented method of  claim 1 , wherein the environment comprises a real-world environment or a simulated environment. 
     
     
       10. An autonomous vehicle control system comprising:
 one or more processors; and 
 one or more computer-readable medium storing instructions that when executed by the one or more processors cause the autonomous vehicle control system to perform operations, the operations comprising:
 (a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities; 
 (b) fusing the sensor data from the at least two different sensor modalities to create a fused sensor sample, wherein a sensor dropout is applied to the at least two different sensor modalities, the sensor dropout including modifying intensity values of at least one of the at least two different sensor modalities; 
 (c) providing the fused sensor sample as input to a machine learning model, the machine learning model having been trained by independently applying sensor dropout to the at least two different sensor modalities; and 
 (d) receiving, as an output of the machine learning model in response to receipt of the fused sensor sample provided as input, data indicative of one or more objects within an environment; and 
 (e) controlling an autonomous vehicle based on the data indicative of the one or more objects within the environment. 
 
 
     
     
       11. The autonomous vehicle control system of  claim 10 , wherein the plurality of sensors comprise a RADAR system, a LIDAR system, and a camera. 
     
     
       12. The autonomous vehicle control system of  claim 11 , wherein:
 the sensor data is Obtained from at least one of the RADAR system or the camera; and 
 a training of the machine learning model comprises zeroing out a final feature vector for a portion of the sensor data obtained from the at least one of the RADAR system or the camera. 
 
     
     
       13. The autonomous vehicle control system of  claim 11 , wherein:
 the sensor data is obtained from the LIDAR system; and 
 a training of the machine learning model comprises replacing a LIDAR intensity value with a sentinel value for a portion of the sensor data obtained from the LIDAR system. 
 
     
     
       14. The autonomous vehicle control system of  claim 10 , wherein the machine learning model comprises an end-to-end model that is configured to jointly perform object detection and motion prediction. 
     
     
       15. An autonomous vehicle comprising:
 one or more processors; and 
 one or more computer-readable medium storing instructions that when executed by the one or more processors cause the autonomous vehicle to perform operations, the operations comprising:
 (a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities; 
 (b) fusing the sensor data from the at least two different sensor modalities to create a fused sensor sample, wherein a sensor dropout is applied to the at least two different sensor modalities, the sensor dropout including modifying intensity values of at least one of the at least two different sensor modalities; 
 (c) providing the fused sensor sample as input to a machine learning model for object detection and motion prediction, the machine learning model for object detection and motion prediction having been trained by independently applying sensor dropout to the at least two different sensor modalities; 
 (d) receiving, as an output of the machine learning model for object detection and motion prediction in response to receipt of the fused sensor sample provided as input, perception data indicative of one or more states of one or more objects within an environment associated with the autonomous vehicle and prediction data indicative of one or more predicted future locations of the one or more objects; 
 (e) generating motion plan data based on the perception data and the prediction data; and 
 (f) controlling the autonomous vehicle based on the motion plan data. 
 
 
     
     
       16. The autonomous vehicle of  claim 15 , wherein the plurality of sensors comprise a RADAR system, a LIDAR system, and a camera. 
     
     
       17. The autonomous vehicle of  claim 16 , wherein:
 the sensor data is obtained from at least one of the RADAR system or the camera; and 
 a training of the machine learning model for object detection and motion prediction comprises zeroing out a final feature vector for a portion of the sensor data Obtained from the at least one of the RADAR system or the camera. 
 
     
     
       18. The autonomous vehicle of  claim 16 , wherein:
 the sensor data is obtained from the LIDAR system; and 
 a training of the machine learning model for object detection and motion prediction comprises replacing a LIDAR intensity value with a sentinel value for a portion of the sensor data obtained from the LIDAR system.