Apparatus for acquiring 3-dimensional maps of a scene

Abstract

An active sensor for performing active measurements of a scene is presented. The active sensor includes at least one transmitter configured to emit light pulses toward at least one target object in the scene, wherein the at least one target object is recognized in an image acquired by a passive sensor; at least one receiver configured to detect light pulses reflected from the at least one target object; a controller configured to control an energy level, a direction, and a timing of each light pulse emitted by the transmitter, wherein the controller is further configured to control at least the direction for detecting each of the reflected light pulses; and a distance measurement circuit configured to measure a distance to each of the at least one target object based on the emitted light pulses and the detected light pulses.

1 - 27 . (canceled) 
     
     
         28 . A machine-readable storage medium encoded with a non-transitory program code for execution by a data processor for generating a 3D map of a scene, the program code configured for:
 a. receiving active sensor data generated by an active sensor performing active measurements of a scene including one or more target objects, including:
 i. a transmitter configured to emit energy pulses toward the scene, 
 ii. a receiver configured to detect energy pulses reflected from the scene to generate in response to the reflected pulses an active sensor representation of the scene including active sensor data conveying a plurality of distance measurements, 
   b. receiving image data from a 2D image sensor, the image data including a plurality of pixels conveying a 2D scene representation,   c. processing the image data to perform image recognition on the 2D scene representation to identify the one or more target objects in the 2D scene representation,   d. fusing the active sensor representation and the 2D scene representation for providing a 3D map of the scene including a succession of data frames, each data frame associating distance measurements from the active sensor data with pixels from the image data.   
     
     
         29 . A machine-readable storage medium as defined in  claim 28 , wherein the program code is further configured for processing the 3D map of the scene to derive directions to operate a vehicle and output the directions to a controller controlling an operation of the vehicle. 
     
     
         30 . A machine-readable storage medium as defined in  claim 28 , wherein the program code is configured for rendering the 3D map. 
     
     
         31 . A machine-readable storage medium as defined in  claim 28 , wherein the active sensor is selected from the group consisting of LIDAR, RADAR and acoustic sensor. 
     
     
         32 . A machine-readable storage medium as defined in  claim 28 , wherein the active sensor data is mapped to the image data wherein the fusing of the active sensor representation and the 2D scene representation includes associating distance measurements of the active sensor data to corresponding pixels of the image data according to the mapping. 
     
     
         33 . A machine-readable storage medium as defined in  claim 28 , wherein the fusing including estimating for one or more pixels, for which no distance values are available in the active sensor data, distance values and associating the estimated distance values to the one or more pixels in the 3D map. 
     
     
         34 . A machine-readable storage medium as defined in  claim 33 , wherein the fusing including, for first pixels in the image data for which no distance measurements are available in the active sensor data, estimating distance values at least in part on a basis of distance values in the active measurement data available for second pixels in the image data that share a common characteristic with the first pixels. 
     
     
         35 . A machine-readable storage medium as defined in  claim 34 , wherein the common characteristic includes pixel color, wherein the first pixels have substantially similar colors than the second pixels. 
     
     
         36 . A machine-readable storage medium as defined in  claim 28 , wherein the program code is further configured for controlling an energy level of pulses emitted by the transmitter, wherein the energy level is one of at least two available non-zero energy levels and wherein the energy level of pulses emitted by the transmitter is based at least in part on a proximity of one or more of the target objects to the active sensor. 
     
     
         37 . A machine-readable storage medium as defined in  claim 36 , wherein the controlling of the energy level including decreasing the energy level with an increasing proximity of the target object to the vehicle. 
     
     
         38 . A machine-readable storage medium as defined in  claim 36 , wherein the controlling of the energy level including increasing the energy level with a decreasing proximity of the target object to the vehicle. 
     
     
         39 . A machine-readable storage medium as defined in  claim 28 , wherein the program code is configured for performing classification on the one or more target objects identified in the 2D scene representation. 
     
     
         40 . A machine-readable storage medium as defined in  claim 39 , wherein the classification includes classifying the one or more target objects in pre-determined categories of objects. 
     
     
         41 . A machine-readable storage medium as defined in  claim 36 , wherein the program code is configured for deriving the proximity of the one or more target objects to the active sensor from the 3D map. 
     
     
         42 . A machine-readable storage medium as defined in  claim 34 , wherein the program code is configured to define segments in the image data based at least in part on the common characteristics, whereby pixels in a given segment share the common characteristic. 
     
     
         43 . A machine-readable storage medium as defined in  claim 42 , wherein the program code is configured to identify three or more distance measurements in the active sensor data associated with pixels in a particular segment and estimates on the basis of the plurality of distance values by using a plane equation, distance values for other pixels in the particular segment for which no distance values are available in the active sensor data. 
     
     
         44 . A machine-readable storage medium encoded with a non-transitory program code for execution by a data processor for generating a 3D map of a scene, the program code configured for:
 a. receiving active sensor data from an active sensor for performing active measurements of a scene including one or more target objects, the active sensor including:
 i. a transmitter configured to emit energy pulses toward the scene, 
 ii. a receiver configured to detect energy pulses reflected from the scene to generate in response to the reflected pulses the active sensor data including a plurality of distance measurements and conveying an active sensor representation of the scene, 
   b. receiving image data generated by a 2D image sensor, including a plurality of pixels, and conveying a 2D scene representation,   c. perform image recognition on the 2D scene representation to identify the one or more target objects in the 2D scene representation,   d. fuse the active sensor representation and the 2D scene representation of the scene for providing a 3D map of the scene including a succession of data frames, each data frame associating distance measurements from the active sensor data with pixels from the image data,   e. derive from the 3D map of the scene a proximity of one or more of the target objects to the active sensor,   f. control at least in part on a basis of the proximity a parameter of the energy pulses emitted by the transmitter.   
     
     
         45 . A machine-readable storage medium as defined in  claim 44 , wherein the parameter of the energy pulses emitted by the transmitter includes energy level, direction, and timing. 
     
     
         46 . A machine-readable storage medium as defined in  claim 44 , wherein the program code is configured for processing the 3D map of the scene to derive vehicle operation directions and output the directions to a controller controlling an operation of the vehicle. 
     
     
         47 . A machine-readable storage medium as defined in  claim 44 , wherein the program code is configured for rendering the 3D map. 
     
     
         48 . A machine-readable storage medium as defined in  claim 43 , wherein the active sensor is selected from the group consisting of LIDAR, RADAR and acoustic sensor. 
     
     
         49 . A machine-readable storage medium as defined in  claim 43 , wherein the active sensor data is mapped to the image data wherein the fusing of the active sensor representation and the 2D scene representation includes associating distance measurements of the active sensor data to corresponding pixels of the image data according to the mapping. 
     
     
         50 . A machine-readable storage medium as defined in  claim 43 , wherein the fusing including estimating for one or more pixels, for which no distance values are available in the active sensor data, distance values and associating the distance values to the one or more pixels in the 3D map. 
     
     
         51 . A machine-readable storage medium as defined in  claim 43 , wherein the program code is configured for, for first pixels in the image data for which no distance measurements are available in the active sensor data, estimating distance values at least in part on a basis of distance values in the active measurement data available for second pixels in the image data that are visually similar to the first pixels. 
     
     
         52 . A machine-readable storage medium as defined in  claim 43 , wherein the computer system is configured for controlling an energy level of pulses emitted by the transmitter, wherein the energy level is one of at least two available non-zero energy levels. 
     
     
         53 . A machine-readable storage medium as defined in  claim 52 , wherein the controlling of the energy level including decreasing the energy level with an increasing proximity of the one or more target objects to the vehicle. 
     
     
         54 . An automated method, comprising:
 a. receiving with one or more processors, first sensor data from a first sensor, the first sensor data representing an environment, the first sensor including a 2D image sensor and the first sensor data including image data providing a 2D representation of the environment,   b. performing with the one or more processors image recognition on the 2D representation of the environment to identify one or more target objects in the 2D representation of the environment,   c. receiving with the one or more processors, a second signal from a second sensor, the second signal including second sensor data representing the environment,   d. performing data fusion between the first sensor data and the second sensor data to produce a succession of data frames, wherein each data frame including data derived from the first sensor data and associated to data derived from the second sensor data to provide a 3D representation of the environment including the one or more target objects,   e. process with the one or more processors the succession of data frames to derive driving directions and output the driving directions to a controller controlling an operation of a vehicle driving in the environment.   
     
     
         55 . An automated method as defined in  claim 54 , wherein the second sensor data conveys distance information, including distance information to the one or more target objects. 
     
     
         56 . An automated method as defined in  claim 55 , wherein the second sensor is selected from the group consisting of LIDAR sensor, RADAR sensor and acoustic sensor. 
     
     
         57 . An automated method as defined in  claim 56 , wherein the image data includes a plurality of pixels, the data fusion including associating in the data frame pixels with distance measurements derived from the second sensor.