Bird's eye view based velocity estimation

Abstract

Systems and methods determining velocity of an object associated with a three-dimensional (3D) scene may include: a LIDAR system generating two sets of 3D point cloud data of the scene from two consecutive point cloud sweeps; a pillar feature network encoding data of the point cloud data to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data sets in the form of pseudo images, wherein the 2D bird's-eye-view embeddings for a first of the two point cloud data sets comprises pillar features for the first point cloud data set and the 2D bird's-eye-view embeddings for a second of the two point cloud data sets comprises pillar features for the second point cloud data set; and a feature pyramid network encoding the pillar features and performing a 2D optical flow estimation to estimate the velocity of the object.

What is claimed is: 
     
       1. A method for determining velocity of an object associated with a three-dimensional (3D) scene, the method comprising:
 receiving two sets of 3D point cloud data of the scene from two consecutive point cloud sweeps; 
 aligning the two consecutive point cloud data sets into the same coordinate frame; 
 encoding data of the point cloud data sets using a pillar feature network to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data sets in the form of pseudo images, wherein the 2D bird's-eye-view embeddings for a first of the two point cloud data sets comprises pillar features for the first point cloud data set and the 2D bird's-eye-view embeddings for a second of the two point cloud data sets comprises pillar features for the second point cloud data set; and 
 encoding the pillar features using a feature pyramid network and performing a 2D optical flow estimation to estimate the velocity of the object. 
 
     
     
       2. The method of  claim 1 , further comprising applying a contextual network to use contextual information to refine the velocity estimate. 
     
     
       3. The method of  claim 2 , wherein the context network is a feedforward CNN based on dilated convolutions. 
     
     
       4. The method of  claim 1 , wherein receiving two sets of 3D point cloud data of the scene comprises receiving the first point cloud data set by a first pillar feature network and receiving a second point cloud data set by a second pillar feature network, wherein the first point cloud data set represents the scene at a time t−1 and the second point cloud data set represents the scene at a time t subsequent to the time t−1. 
     
     
       5. The method of  claim 1 , wherein encoding data of the point cloud data sets comprises voxelizing the point cloud data sets to render surfaces in the data sets onto a grid of discretized volume elements in a 3D space to create a set of pillars. 
     
     
       6. The method of  claim 5 , wherein the set of pillars comprise a (D, P, N) shape tensor in which P is the number of pillars and N denotes the number of points per pillar. 
     
     
       7. The method of  claim 5 , further comprising encoding voxel information from the voxelizing to extract the features of the point cloud data sets. 
     
     
       8. The method of  claim 7 , further comprising scattering the encoded features back to their original pillar locations to create the bird's-eye-view. 
     
     
       9. The method of  claim 1 , wherein the 2D optical flow estimation comprises warping the pseudo image of the first point cloud data set to align the pseudo image of the first point cloud data set with the pseudo image of the second point cloud data set. 
     
     
       10. The method of  claim 9 , wherein the 2D optical flow estimation further comprises computing a cost function of the warped pseudo image of the first point cloud data set and the pseudo image of the second point cloud data set, by identifying displacement of a feature from the first image to the second image. 
     
     
       11. The method of  claim 10 , wherein the 2D optical flow estimation further comprises using the cost function to estimate the flow of the object. 
     
     
       12. The method of  claim 1 , wherein performing a 2D optical flow estimation to estimate the velocity of the object comprises aggregating bird's eye view motion vectors to compute a single mean velocity and co-variance for each obstacle cluster. 
     
     
       13. The method of  claim 12 , wherein a sample is weighted based on an occupancy probability of the cell to which the sample belongs. 
     
     
       14. The method of  claim 1 , wherein estimated velocity of the object is a 2-D flow vector for the object. 
     
     
       15. The method of  claim 1 , further comprising using annotated track cuboids to auto-generate the 2D flow in multiple scales. 
     
     
       16. The method of  claim 1 , further comprising performing flow estimation only on labeled dynamic objects and not performing flow estimation on non-labeled obstacles or background objects. 
     
     
       17. The method of  claim 1 , wherein the method is performed using three or more sets of 3D point cloud data of the scene, including aligning all of the point cloud data sets into the same coordinate frame, encoding data of each of the point cloud data sets using a pillar feature network to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data sets comprising pillar features for each point cloud data set, and encoding the pillar features using a feature pyramid network and performing a 2D optical flow estimation to estimate the velocity of the object. 
     
     
       18. The method of  claim 1 , wherein encoding the pillar features using a feature pyramid network further includes using 2D map information as an additional channel input to the feature pyramid network. 
     
     
       19. The method of  claim 1 , further comprising filtering the point cloud datasets using a ground height map, wherein the filtering comprises comparing data point heights against ground height and discarding a data point whose point height is not greater than the ground height at the point's location. 
     
     
       20. A system for determining velocity of an object associated with a three-dimensional (3D) scene, the system comprising:
 a non-transitory memory configured to store instructions; 
 at least one processor configured to execute the instructions to perform the operations of:
 receiving two sets of 3D point cloud data of the scene from two consecutive point cloud sweeps; 
 aligning the two consecutive point cloud data sets into the same coordinate frame; 
 encoding data of the point cloud data sets using a pillar feature network to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data sets in the form of pseudo images, wherein the 2D bird's-eye-view embeddings for a first of the two point cloud data sets comprises pillar features for the first point cloud data set and the 2D bird's-eye-view embeddings for a second of the two point cloud data sets comprises pillar features for the second point cloud data set; and 
 encoding the pillar features using a feature pyramid network and performing a 2D optical flow estimation to estimate the velocity of the object.