Systems and methods for detecting unknown objects on a road surface by an autonomous vehicle

Abstract

An autonomous vehicle comprises one or more processors. The processors can be configured to receive, from a sensor of the autonomous vehicle, an image of an environment outside of the autonomous vehicle. The processors can detect potential unknown objects based on the image. The processors can compare the detection based on the image to a set of data points of a LiDAR scan to determine if there are unknown objects on a roadway.

What is claimed is: 
     
         1 . A method comprising:
 receiving, by a processor, from a sensor of an autonomous vehicle, an image of an environment outside of the autonomous vehicle;   identifying, by the processor, a mask image for a road surface and an unknown object using a panoptic segmentation model;   extracting, by the processor, a two-dimensional bounding box for the unknown object;   retrieving, by the processor, a set of data points received from a LiDAR sensor of the autonomous vehicle monitoring the environment of the autonomous vehicle;   identifying, by the processor, a first subset of the set of data points corresponding to foreground data points and a second subset of the set of data points corresponding to background data points; and   generating, by the processor, using the mask image and the first subset of the set of data points, a three-dimensional bounding box for the unknown object.   
     
     
         2 . The method of  claim 1 , further comprising:
 predicting, by the processor, a ground plane for the set of data points received from the LiDAR sensor.   
     
     
         3 . The method of  claim 1 , further comprising:
 executing, by the processor, an object recognition model using the three-dimensional bounding box for the unknown object.   
     
     
         4 . The method of  claim 1 , further comprising:
 projecting, by the processor, the set of data points onto the mask image.   
     
     
         5 . The method of  claim 4 , wherein the processor projects the set of data points in a region associated with the unknown object. 
     
     
         6 . A system comprising:
 one or more processors, wherein the one or more processors are configured to execute instructions on a non-transitory computer-readable medium to:
 receive, from a sensor of an autonomous vehicle, an image of an environment outside of the autonomous vehicle; 
 identify a mask image for a road surface and an unknown object using a panoptic segmentation model; 
 extract a two-dimensional bounding box for the unknown object; 
 retrieve a set of data points received from a LiDAR sensor of the autonomous vehicle monitoring the environment of the autonomous vehicle; 
 identify a first subset of the set of data points corresponding to foreground data points and a second subset of the set of data points corresponding to background data points; and 
 generate, using the mask image and the first subset of the set of data points, a three-dimensional bounding box for the unknown object. 
   
     
     
         7 . The system of  claim 6 , wherein the instructions further cause the one or more processors to:
 predict a ground plane for the set of data points received from the LiDAR sensor.   
     
     
         8 . The system of  claim 6 , wherein the instructions further cause the one or more processors to:
 execute an object recognition model using the three-dimensional bounding box for the unknown object.   
     
     
         9 . The system of  claim 6 , wherein the instructions further cause the one or more processors to:
 project the set of data points onto the mask image.   
     
     
         10 . The system of  claim 9 , wherein the one or more processors project the set of data points in a region associated with the unknown object. 
     
     
         11 . A non-transitory computer readable medium including one or more instructions stored thereon and executable by a processor to:
 receive, from a sensor of an autonomous vehicle, an image of an environment outside of the autonomous vehicle;   identify a mask image for a road surface and an unknown object using a panoptic segmentation model;   extract a two-dimensional bounding box for the unknown object;   retrieve a set of data points received from a LiDAR sensor of the autonomous vehicle monitoring the environment of the autonomous vehicle;   identify a first subset of the set of data points corresponding to foreground data points and a second subset of the set of data points corresponding to background data points; and   generate, using the mask image and the first subset of the set of data points, a three-dimensional bounding box for the unknown object.   
     
     
         12 . The non-transitory computer readable medium of  claim 11 , wherein the one or more instructions executable by the processor to:
 predict a ground plane for the set of data points received from the LiDAR sensor.   
     
     
         13 . The non-transitory computer readable medium of  claim 11 , wherein the one or more instructions executable by the processor to:
 execute an object recognition model using the three-dimensional bounding box for the unknown object.   
     
     
         14 . The non-transitory computer readable medium of  claim 11 , wherein the one or more instructions executable by the processor to:
 project the set of data points onto the mask image.