Calibration between depth and color sensors for depth cameras

Abstract

A system described herein includes a receiver component that receives a first digital image from a color camera, wherein the first digital image comprises a planar object, and a second digital image from a depth sensor, wherein the second digital image comprises the planar object. The system also includes a calibrator component that jointly calibrates the color camera and the depth sensor based at least in part upon the first digital image and the second digital image.

What is claimed is: 
     
       1. A method, comprising:
 receiving an image of a scene generated by a color camera, the scene comprising a planar object, the planar object having a known pattern; 
 locating, in the image, the pattern in the planar object based upon the pattern in the planar object being known; 
 computing a three-dimensional position and orientation of the planar object in the scene responsive to the pattern being located in the image, the three-dimensional position and orientation of the planar object derived from the image of the scene generated by the color camera; 
 receiving a depth image of the scene generated by a depth sensor, the depth image comprises pixels having depth values; and 
 jointly calibrating the color camera and the depth sensor based upon:
 the computed three-dimensional position and orientation of the planar object; and 
 the depth values of the pixels in the depth image generated by the depth sensor. 
 
 
     
     
       2. The method of  claim 1 , wherein the color camera has a first coordinate system and the depth sensor has a second coordinate system, and wherein jointly calibrating the color camera and the depth sensor comprises determining a rotation and translation between the first coordinate system and the second coordinate system. 
     
     
       3. The method of  claim 2 , wherein jointly calibrating the color camera and the depth sensor comprises calculating a plurality of values of intrinsic parameters of the color camera and the depth sensor, the plurality of intrinsic parameters comprising a focus, a camera center, and a depth mapping function. 
     
     
       4. The method of  claim 1 , further comprising:
 receiving a first plurality of images of the scene that are generated by the color camera over time; 
 receiving a second plurality of images of the scene that are generated by the depth sensor over time, wherein the planar object is at different locations in the scene relative to the color camera and the depth sensor in each of the images in the first plurality of images and the second plurality of images; and 
 jointly calibrating the color camera and the depth sensor based upon the first plurality of images and the second plurality of images. 
 
     
     
       5. The method of  claim 1 , wherein the color camera is a video camera and the depth sensor comprises an infrared camera. 
     
     
       6. The method of  claim 1 , wherein the depth sensor is one of a time of flight sensor or a structured light sensor. 
     
     
       7. The method of  claim 1 , wherein the planar object is a checkerboard, the known pattern being a checkerboard pattern. 
     
     
       8. The method of  claim 1 , wherein jointly calibrating the color camera and the depth sensor further comprises
 fitting a plane on the depth image based upon the computed three-dimensional position and orientation of the planar object; and 
 learning a translation and rotation between a coordinate system of the depth sensor and a coordinate system of the color camera based upon an estimated correspondence between the computed three-dimensional position and orientation of the planar object and the plane fitted on the depth image. 
 
     
     
       9. The method of  claim 1 , wherein jointly calibrating the color camera and the depth sensor comprises:
 sampling the values of the pixels in the depth image; and 
 learning a likelihood function that is configured to output a likelihood that a particular pixel in the depth image corresponds to the planar object. 
 
     
     
       10. The method of  claim 9 , wherein jointly calibrating the color camera and the depth sensor further comprises learning a translation and rotation between a coordinate system of the depth sensor and a coordinate system of the color camera based upon an evaluation of the likelihood function. 
     
     
       11. The method of  claim 1 , further comprising:
 subsequent to jointly calibrating the color camera and the depth sensor, receiving a first image from the color camera; 
 subsequent to jointly calibrating the color camera and the depth sensor, receiving a second image from the depth sensor; and 
 overlaying at least a portion of the first image onto the second image to generate a three- dimensional image based upon the calibrating of the color camera and the depth sensor. 
 
     
     
       12. A system comprising:
 a receiver component that receives:
 a first digital image of a scene from a color camera, wherein the scene comprises a planar object that has a known pattern; and 
 a second digital image of the scene from a depth sensor, the first digital image and the second digital image being coincident in time; and 
 a calibrator component that jointly calibrates the color camera and the depth sensor based upon:
 a computed position and orientation of the planar object in a coordinate system of the first digital image; and 
 values of pixels in the second digital image that correspond to the planar object. 
 
 
 
     
     
       13. The system of  claim 12  comprised by a gaming console. 
     
     
       14. The system of  claim 12 , wherein the color camera and the depth sensor are included together in a housing. 
     
     
       15. The system of  claim 12 , wherein the planar object is a checkerboard. 
     
     
       16. The system of  claim 12 , wherein the calibrator component outputs a rotation and translation between the coordinate system of the color camera and a coordinate system of the depth sensor. 
     
     
       17. The system of  claim 16 , further comprising:
 a mapper component that maps pixels of an image generated by the color camera to pixels of an image generated by the depth sensor. 
 
     
     
       18. The system of  claim 17 , wherein the mapper component generates a three-dimensional image based upon the pixels of the image generated by the color camera being mapped to the pixels of the image generated by the depth sensor. 
     
     
       19. A computer-readable data storage medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:
 outputting at least one instruction to a user with respect to placement of a planar object having a known pattern relative to a color camera and a depth sensor; 
 subsequent to outputting the at least one instruction, causing the color camera to capture a first image of a scene that includes the planar object; 
 causing the depth sensor to capture a second image of the scene, the first image and the second image being coincident in time; 
 computing a position and orientation of the planar object in a coordinate system of the color camera; 
 identifying a plane that includes the planar object in the coordinate system of the color camera, the plane identified based upon the computed position and orientation of the planar object in the coordinate system of the color camera; 
 identifying pixels in the second image that correspond to the planar object based upon values of the pixels; 
 fitting the plane in the second image based upon the values of the pixels; and 
 computing an estimated translation and rotation between the coordinate system of the color camera and a coordinate system of the depth sensor based upon the fitting of the plane in the second image. 
 
     
     
       20. The computer-readable data storage medium of  claim 19 , the acts further comprising:
 subsequent to computing the estimated translation and rotation between the coordinate system of the color camera and the coordinate system of the depth sensor, generating a three-dimensional color image based upon the estimated translation and rotation.