Using light detection and ranging (LIDAR) to train camera and imaging radar deep learning networks

Abstract

Disclosed are techniques for annotating image frames using information from a light detection and ranging (LiDAR) sensor. An exemplary method includes receiving, from the LiDAR sensor, at least one LiDAR frame, receiving, from a camera sensor, at least one image frame, removing LiDAR points that represent a ground surface of the environment, identifying LiDAR points of interest in the at least one LiDAR frame, segmenting the LiDAR points of interest to identify at least one object of interest in the at least one LiDAR frame, and annotating the at least one image frame with a three-dimensional oriented bounding box of the at least one object of interest detected in the at least one image frame by projecting the three-dimensional oriented bounding boxes from the at least one LiDAR frame to the at least one image frame using cross-calibration transforms between the LiDAR sensor and the camera.

What is claimed is: 
     
       1. A method for annotating image frames using information from a light detection and ranging (LiDAR) sensor, comprising:
 receiving, from the LiDAR sensor of a first vehicle, at least one LiDAR frame representing an environment of the first vehicle, the at least one LiDAR frame comprising a plurality of LiDAR points; 
 receiving, from a camera sensor or an imaging radar sensor, at least one image frame of the environment of the first vehicle; 
 removing LiDAR points from the at least one LiDAR frame that represent a ground surface of the environment of the first vehicle; 
 identifying LiDAR points of interest in the at least one LiDAR frame; 
 segmenting the LiDAR points of interest to identify at least one object of interest in the at least one LiDAR frame; and 
 annotating the at least one image frame with a three-dimensional oriented bounding box of the at least one object of interest detected in the at least one image frame by projecting the three-dimensional oriented bounding box from the at least one LiDAR frame to the at least one image frame using cross-calibration transforms between the LiDAR sensor and the camera sensor or between the LiDAR sensor and the imaging radar sensor. 
 
     
     
       2. The method of  claim 1 , further comprising:
 training an object-detection neural network to detect the at least one object of interest in the radar frame or the image frame based on the three-dimensional oriented bounding box. 
 
     
     
       3. The method of  claim 2 , wherein the object-detection neural network is implemented by a processor of a vehicle that does not have a LiDAR sensor. 
     
     
       4. The method of  claim 1 , further comprising:
 tracking the at least one object of interest across the at least one LiDAR frame and at least one subsequent LiDAR frame. 
 
     
     
       5. The method of  claim 4 , wherein tracking the at least one object of interest comprises:
 determining dimensions of the at least one object of interest when the at least one object of interest is within a threshold distance of the first vehicle. 
 
     
     
       6. The method of  claim 1 , wherein segmenting the LiDAR points of interest comprises:
 creating an adjacency graph representing the LiDAR points of interest in the at least one LiDAR frame; and 
 identifying a plurality of connected components within the adjacency graph as corresponding to the at least one object of interest. 
 
     
     
       7. The method of  claim 1 , wherein the image frame comprises an imaging radar image frame. 
     
     
       8. The method of  claim 1 , wherein the image frame comprises a camera image frame. 
     
     
       9. The method of  claim 1 , wherein the camera sensor captures the at least one image frame of the environment of the first vehicle at a corresponding point in time as the LiDAR frame. 
     
     
       10. The method of  claim 1 , wherein the at least one object of interest comprises at least one vehicle. 
     
     
       11. The method of  claim 1 , further comprising:
 receiving, from a LiDAR sensor of a second vehicle, at least one second LiDAR frame representing an environment of the second vehicle, the at least one second LiDAR frame comprising a second plurality of LiDAR points; 
 receiving, from a camera sensor or an imaging radar sensor of the second vehicle, at least one second image frame of the environment of the second vehicle; 
 removing LiDAR points from the at least one second LiDAR frame that represent a ground surface of the environment of the second vehicle; 
 identifying second LiDAR points of interest in the at least one second LiDAR frame; 
 segmenting the second LiDAR points of interest to identify at least one second object of interest in the at least one second LiDAR frame; and 
 annotating the at least one second image frame with a second three-dimensional oriented bounding box of the at least one second object of interest detected in the at least one second image frame by projecting the second three-dimensional oriented bounding box from the at least one second LiDAR frame to the at least one second image frame using cross-calibration transforms between the LiDAR sensor and the camera sensor or the imaging radar sensor of the second vehicle. 
 
     
     
       12. The method of  claim 11 , wherein the at least one image frame is further annotated with the second three-dimensional oriented bounding box of the at least one second object of interest by projecting the second three-dimensional oriented bounding box from the at least one second LiDAR frame to the at least one image frame using cross-calibration transforms between the LiDAR sensor of the second vehicle and the camera sensor or the imaging radar sensor of the first vehicle. 
     
     
       13. A method performed by at least one processor of an ego vehicle, comprising:
 receiving, from an imaging radar sensor of the ego vehicle, at least one radar frame of the environment of the ego vehicle; 
 detecting at least one object of interest in the at least one radar frame; 
 determining a three-dimensional oriented bounding box, a location, and a velocity of the at least one object of interest; and 
 outputting the three-dimensional oriented bounding box, the location, and the velocity of the at least one object of interest, wherein an object-detection neural network is used to detect the at least one object of interest in the at least one radar frame and the object-detection neural network is trained using radar frame data annotated with light detection and ranging (LiDAR) data. 
 
     
     
       14. The method of  claim 13 , wherein the at least one object of interest comprises at least one vehicle. 
     
     
       15. An apparatus for annotating image frames using information from a light detection and ranging (LiDAR) sensor, comprising:
 at least one processor configured to:
 receive, from the LiDAR sensor of a first vehicle, at least one LiDAR frame representing an environment of the first vehicle, the at least one LiDAR frame comprising a plurality of LiDAR points; 
 receive, from a camera sensor or an imaging radar sensor, at least one image frame of the environment of the first vehicle; 
 remove LiDAR points from the at least one LiDAR frame that represent a ground surface of the environment of the first vehicle; 
 identify LiDAR points of interest in the at least one LiDAR frame; 
 segment the LiDAR points of interest to identify at least one object of interest in the at least one LiDAR frame; and 
 annotate the at least one image frame with a three-dimensional oriented bounding box of the at least one object of interest detected in the at least one image frame by projecting the three-dimensional oriented bounding box from the at least one LiDAR frame to the at least one image frame using cross-calibration transforms between the LiDAR sensor and the camera sensor or between the LiDAR sensor and the imaging radar sensor. 
 
 
     
     
       16. The apparatus of  claim 15 , wherein the at least one processor is further configured to:
 train an object-detection neural network to detect the at least one object of interest in the radar frame or the image frame based on the three-dimensional oriented bounding box. 
 
     
     
       17. The apparatus of  claim 15 , wherein the at least one processor is further configured to:
 track the at least one object of interest across the at least one LiDAR frame and at least one subsequent LiDAR frame. 
 
     
     
       18. The apparatus of  claim 17 , wherein the at least one processor being configured to track the at least one object of interest comprises the at least one processor being configured to:
 determine dimensions of the at least one object of interest when the at least one object of interest is within a threshold distance of the first vehicle. 
 
     
     
       19. The apparatus of  claim 15 , wherein the at least one processor being configured to segment the LiDAR points of interest comprises the at least one processor being configured to:
 create an adjacency graph representing the LiDAR points of interest in the at least one LiDAR frame; and 
 identify a plurality of connected components within the adjacency graph as corresponding to the at least one object of interest. 
 
     
     
       20. The apparatus of  claim 15 , wherein the image frame comprises an imaging radar image frame. 
     
     
       21. The apparatus of  claim 15 , wherein the image frame comprises a camera image frame. 
     
     
       22. The apparatus of  claim 15 , wherein the camera sensor captures the at least one image frame of the environment of the first vehicle at a corresponding point in time as the LiDAR frame. 
     
     
       23. The apparatus of  claim 15 , wherein the at least one processor is further configured to:
 receive, from a LiDAR sensor of a second vehicle, at least one second LiDAR frame representing an environment of the second vehicle, the at least one second LiDAR frame comprising a second plurality of LiDAR points; 
 receive, from a camera sensor or an imaging radar sensor of the second vehicle, at least one second image frame of the environment of the second vehicle; 
 remove LiDAR points from the at least one second LiDAR frame that represent a ground surface of the environment of the second vehicle; 
 identify second LiDAR points of interest in the at least one second LiDAR frame; 
 segment the second LiDAR points of interest to identify at least one second object of interest in the at least one second LiDAR frame; and 
 annotate the at least one second image frame with a second three-dimensional oriented bounding box of the at least one second object of interest detected in the at least one second image frame by projecting the second three-dimensional oriented bounding box from the at least one second LiDAR frame to the at least one second image frame using cross-calibration transforms between the LiDAR sensor and the camera sensor or the imaging radar sensor of the second vehicle. 
 
     
     
       24. The apparatus of  claim 23 , wherein the at least one image frame is further annotated with the second three-dimensional oriented bounding box of the at least one second object of interest by projecting the second three-dimensional oriented bounding box from the at least one second LiDAR frame to the at least one image frame using cross-calibration transforms between the LiDAR sensor of the second vehicle and the camera sensor or the imaging radar sensor of the first vehicle. 
     
     
       25. An ego vehicle, comprising:
 at least one processor configured to:
 receive, from an imaging radar sensor of the ego vehicle, at least one radar frame of the environment of the ego vehicle; 
 detect at least one object of interest in the at least one radar frame; 
 determine a three-dimensional oriented bounding box, a location, and a velocity of the at least one object of interest; and 
 output the three-dimensional oriented bounding box, the location, and the velocity of the at least one object of interest, wherein an object-detection neural network is used to detect the at least one object of interest in the at least one radar frame and the object-detection neural network is trained using radar frame data annotated with light detection and ranging (LiDAR) data. 
 
 
     
     
       26. The ego vehicle of  claim 25 , wherein the at least one object of interest comprises at least one vehicle.