3D vision system with automatically calibrated stereo vision sensors and LiDAR sensor

Abstract

An automatically calibrated vision system includes: a vision calibration system; a first sensor system that receives first data of a scene captured by a first sensor on a movable machine and outputs a first map based on the first data; a second sensor system that receives second data of the scene captured by a second sensor on the movable machine and outputs a second map based on the second data. The vision calibration system computes calibration data based on the first and second maps, supplies the calibration data to the first sensor system and/or the second sensor system for automatic calibration of the first sensor system and/or the second sensor system, and outputs, to a controller of the movable machine, a calibrated depth map comprised of depth measurements calibrated using the calibration data. The first sensor system and/or the second sensor system automatically perform(s) a self-calibration using the calibration data.

What is claimed is: 
     
       1. An automatically calibrated vision system for an autonomous movable machine, the vision system comprising:
 a vision calibration system; 
 a first sensor system configured to process stereo image data of a scene captured by first and second camera sensors on the movable machine and to output a first map based on the stereo image data; and 
 a second sensor system configured to process lidar data of the scene captured by a lidar sensor on the movable machine and to output a second map based on the lidar data, 
 wherein the first and second camera sensors and the lidar sensor are synchronized such that the stereo image data and the lidar data are captured simultaneously, 
 wherein the vision calibration system is configured to:
 compute calibration data by performing a lidar/stereo calibration based on the first and second maps, the lidar/stereo calibration simultaneously minimizing errors based on the first map and errors based on the second map, 
 supply the calibration data to the first sensor system and the second sensor system for automatic calibration of the first sensor system and automatic calibration of the second sensor system, and 
 output in real time or nearly real time, to a controller of the movable machine, a calibrated depth map comprising depth measurements calibrated using the calibration data, and 
 
 wherein the first sensor system and the second sensor system automatically perform self-calibration using the calibration data, the self-calibration comprising:
 calibrating a rotation parameter R of the lidar sensor relative to the first camera sensor, 
 calibrating a translation parameter T of the lidar sensor relative to the first camera sensor, and 
 calibrating a stereo yaw angle of the first and second camera sensors. 
 
 
     
     
       2. The vision system of  claim 1 , wherein:
 the first map is a depth map, 
 the second map is a point cloud, and 
 the vision calibration system computes the calibration data by:
 generating an enhanced image based assigning distance measurements of pixels of the depth map to pixels of a rectified image of the stereo image data, 
 projecting the point cloud onto the enhanced image, and 
 determining differences between the point cloud and the enhanced image. 
 
 
     
     
       3. The vision system of  claim 2 , wherein the calibration data is computed based on an average of the differences between the point cloud and the enhanced image. 
     
     
       4. The vision system of  claim 2 , wherein:
 the vision calibration system computes the calibration data by converting the differences between the point cloud and the enhanced image into disparities, and 
 the calibration data is computed based on an average of the disparities. 
 
     
     
       5. The vision system of  claim 1 , wherein the self-calibration comprises calibrating a rotation parameter R of the first and second camera sensors and calibrating a translation parameter T of the first and second camera sensors. 
     
     
       6. The vision system of  claim 1 , wherein the self-calibration comprises calibrating a rotation parameter R of the first camera sensor relative to the second stereo camera sensor and/or calibrating a translation parameter T of the first camera sensor relative to the second stereo camera sensor. 
     
     
       7. The vision system of  claim 2 , wherein the vision calibration system is configured to:
 accumulate a plurality of sets of differences for a plurality of enhanced images and a plurality of point clouds determined from a plurality of sets of lidar data and a plurality of sets of stereo image data, and 
 compute the calibration data based on an average of the plurality of sets of differences. 
 
     
     
       8. A non-transitory computer-readable storage medium storing computer-executable code that, when executed by a processing system comprising at least one computer processor, causes the processing system to automatically perform a calibration method to calibrate a vision system for an autonomous movable machine, the calibration method is-comprising:
 computing calibration data by performing a lidar/stereo calibration based on a first map obtained from a stereo vision system and a second map obtained from a lidar system, wherein:
 the first map corresponds to stereo image data captured by first and second camera sensors of the stereo vision system, 
 the second map corresponds to lidar data captured by a lidar sensor of the lidar system, 
 the stereo image data and the lidar data are captured simultaneously, and 
 the lidar/stereo calibration simultaneously minimizes errors based on the first map and errors based on the second map; 
 
 outputting the calibration data to to the stereo vision system and the lidar system for automatic self-calibration of the stereo vision system and the lidar system in real time or nearly real time using the calibration data, the self-calibration comprising:
 calibrating a rotation parameter R of the lidar sensor relative to the first camera sensor, 
 calibrating a translation parameter T of the lidar sensor relative to the first camera sensor, and 
 calibrating a stereo yaw angle of the first and second camera sensors of the stereo vision system; and 
 
 outputting in real time or nearly real time, to a controller of the movable machine, a calibrated depth map comprising depth measurements calibrated using the calibration data. 
 
     
     
       9. The computer-readable storage medium of  claim 8 ,
 wherein:
 the first map is a depth map, 
 the second map is a point cloud, and 
 
 wherein the calibration data is computed by:
 generating an enhanced image based assigning distance measurements of pixels of the depth map to pixels of a rectified image of the stereo image data, 
 projecting the point cloud onto the enhanced image, and 
 determining differences between the point cloud and the enhanced image. 
 
 
     
     
       10. The computer-readable storage medium of  claim 9 , wherein the calibration data is computed based on an average of the differences between the point cloud and the enhanced image. 
     
     
       11. The computer-readable storage medium of  claim 9 , wherein:
 the calibration data is computed by converting the differences between the point cloud and the enhanced image into disparities, and 
 the calibration data is based on an average of the disparities. 
 
     
     
       12. The computer-readable storage medium of  claim 9 , wherein the calibration method further comprises:
 accumulating a plurality of sets of differences for a plurality of enhanced images and a plurality of point clouds determined from a plurality of sets of lidar data and a plurality of sets of stereo image data, and 
 computing the calibration data based on an average of the plurality of sets of differences.