Systems and methods for multi-resolution fusion of pseudo-LiDAR features

Abstract

The embodiments disclosed herein describe vehicles, systems and methods for multi-resolution fusion of pseudo-LiDAR features. In one aspect, a method for multi-resolution fusion of pseudo-LiDAR features includes receiving image data from one or more image sensors, generating a point cloud from the image data, generating, from the point cloud, a first bird's eye view map having a first resolution, generating, from the point cloud, a second bird's eye view map having a second resolution, and generating a combined bird's eye view map by combining features of the first bird's eye view map with features from the second bird's eye view map.

The invention claimed is: 
     
       1. A method for multi-resolution fusion of pseudo-LiDAR features, the method comprising:
 receiving image data from one or more image sensors; 
 generating a point cloud from the image data, wherein the point cloud comprises pseudo-LiDAR data; 
 generating, from the point cloud, a first bird's eye view map having a first resolution; 
 generating, from the point cloud, a second bird's eye view map having a second resolution; and 
 generating a combined bird's eye view map by combining features of the first bird's eye view map with features from the second bird's eye view map. 
 
     
     
       2. The method of  claim 1 , further comprising generating, from the point cloud, one or more additional bird's eye view maps having one or more additional resolutions, wherein generating the combined bird's eye view map further comprises combining features of the one or more additional bird's eye view maps with the features of the first bird's eye view map and the second bird's eye view map. 
     
     
       3. The method of  claim 1 , wherein:
 generating the first bird's eye view map comprises:
 subdividing the pseudo-LiDAR data into a plurality of voxels having a first volume that is determined by the first resolution; and 
 generating an array of first cells comprising one or more layers by extracting one or more features from the pseudo-LiDAR data within the plurality of voxels, wherein each layer of the one or more layers comprises a plurality of pixels, each pixel comprises an individual feature extracted from an individual voxel, and a number of layers equals a number of features extracted from the pseudo-LiDAR data; 
 
 generating the second bird's eye view map comprises:
 subdividing the pseudo-LiDAR data into a plurality of voxels having a second volume that is determined by the first resolution; and 
 generating an array of second cells comprising one or more layers by extracting one or more features from the pseudo-LiDAR data within the plurality of voxels, wherein each layer of the one or more layers comprises a plurality of pixels, each pixel comprises an individual feature extracted from an individual voxel, and a number of layers equals a number of features extracted from the pseudo-LiDAR data. 
 
 
     
     
       4. The method of  claim 3 , wherein the generating of the combined bird's eye view map comprises combining the one or more features from each first cell with the one or more features from each second cell that align with first cells. 
     
     
       5. The method of  claim 1 , wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by an addition function. 
     
     
       6. The method of  claim 1 , wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by a neural network. 
     
     
       7. The method of  claim 1 , wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by concatenation. 
     
     
       8. A method of detecting an object, the method comprising:
 generating a bird's eye view of pseudo-LiDAR data by:
 receiving image data from one or more image sensors; 
 generating a point cloud from the image data, wherein the point cloud comprises pseudo-LiDAR data; 
 generating, from the point cloud, a first bird's eye view map having a first resolution; 
 generating, from the point cloud, a second bird's eye view map having a second resolution; and 
 generating a combined bird's eye view map by combining features of the first bird's eye view map with features from the second bird's eye view map; and 
 
 detecting, using an object detection algorithm, one or more objects from the combined bird's eye view map. 
 
     
     
       9. The method of  claim 8 , further comprising generating, from the point cloud, one or more additional bird's eye view maps having one or more additional resolutions, wherein generating the combined bird's eye view map further comprises combining features of the one or more additional bird's eye view maps with the features of the first bird's eye view map and the second bird's eye view map. 
     
     
       10. The method of  claim 8 , wherein:
 generating the first bird's eye view map comprises:
 subdividing the pseudo-LiDAR data into a plurality of voxels having a first volume that is determined by the first resolution; and 
 generating an array of first cells comprising one or more layers by extracting one or more features from the pseudo-LiDAR data within the plurality of voxels, wherein each layer of the one or more layers comprises a plurality of pixels, each pixel comprises an individual feature extracted from an individual voxel, and a number of layers equals a number of features extracted from the pseudo-LiDAR data; 
 
 generating the second bird's eye view map comprises:
 subdividing the pseudo-LiDAR data into a plurality of voxels having a second volume that is determined by the first resolution; and 
 generating an array of second cells comprising one or more layers by extracting one or more features from the pseudo-LiDAR data within the plurality of voxels, wherein each layer of the one or more layers comprises a plurality of pixels, each pixel comprises an individual feature extracted from an individual voxel, and a number of layers equals a number of features extracted from the pseudo-LiDAR data. 
 
 
     
     
       11. The method of  claim 10 , wherein the generating of the combined bird's eye view map comprises combining the one or more features from each first cell with the one or more features from each second cell that align with first cells in a vertical direction. 
     
     
       12. The method of  claim 8 , wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by an addition function. 
     
     
       13. The method of  claim 8 , wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by a neural network. 
     
     
       14. The method of  claim 8 , wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by concatenation. 
     
     
       15. The method of  claim 8 , wherein the object detection algorithm comprises an object detection neural network. 
     
     
       16. A vehicle comprising:
 one or more image sensors that produce image data; 
 one or more processors; 
 one or memory modules comprising a non-transitory computer readable memory storing computer readable instructions that, when executed by the one or more processors, cause the one or more processors to:
 receive the image data from the one or more image sensors; 
 generate a point cloud from the image data, wherein the point cloud comprises pseudo-LiDAR data; 
 generate, from the point cloud, a first bird's eye view map having a first resolution; 
 generate, from the point cloud, a second bird's eye view map having a second resolution; 
 generate a combined bird's eye view map by combining features of the first bird's eye view map with features from the second bird's eye view map; and 
 detect, using an object detection algorithm, one or more objects from the combined bird's eye view map. 
 
 
     
     
       17. The vehicle of  claim 16 , wherein the computer readable instructions further cause the one or more processors to control a movement of the vehicle based at least in part on the detected one or more objects. 
     
     
       18. The vehicle of  claim 16 , wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by an addition function. 
     
     
       19. The vehicle of  claim 16 , wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by a neural network. 
     
     
       20. The vehicle of  claim 16 , wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by concatenation.