Methods for automatic registration of 3d image data

Abstract

A method for automatic registration of 3D image data, captured by a 3D image capture system having an RGB camera and a depth camera, includes capturing 2D image data with the RGB camera at a first pose; capturing depth data with the depth camera at the first pose; performing an initial registration of the RGB camera to the depth camera; capturing 2D image data with the RGB camera at a second pose; capturing depth data at the second pose; and calculating an updated registration of the RGB camera to the depth camera.

We claim: 
     
         1 . A method for automatic registration of 3D image data captured by a 3D image capture system having an RGB camera and a depth camera, the method comprising:
 capturing 2D image data with the RGB camera at a first pose;   capturing depth data with the depth camera at the first pose;   performing an initial registration of the RGB camera to the depth camera;   capturing 2D image data with the RGB camera at a second pose;   capturing depth data at the second pose; and   calculating an updated registration of the RGB camera to the depth camera.   
     
     
         2 . The method of  claim 1 , wherein calculating the updated registration comprises:
 identifying a set of points common to the depth data captured at the first pose and the depth data captured at the second pose;   calculating a color/texture error function for the set of points; and   updating registration data to reduce the color/texture error function.   
     
     
         3 . The method of  claim 2 , wherein the depth camera is a structured light depth camera; wherein capturing depth data with the depth camera at the first pose comprises capturing depth data using a structured light technique. 
     
     
         4 . The method of  claim 3 , wherein capturing depth data at the second pose comprises capturing depth data with the depth camera using a structured light technique. 
     
     
         5 . The method of  claim 3 , further comprising tracking motion of the 3D image capture system between the first and second poses; wherein capturing depth data at the second pose comprises capturing depth data using a derived data technique; wherein using the derived data technique comprises calculating depth data at the second pose from the depth data captured at the first pose and the tracked motion. 
     
     
         6 . The method of  claim 5 , wherein tracking motion comprises tracking motion using an inertial measurement unit. 
     
     
         7 . The method of  claim 5 , wherein tracking motion comprises tracking motion using an iterative closest point tracking technique. 
     
     
         8 . The method of  claim 2 , wherein the depth camera is a stereo depth camera system comprising two infrared cameras; wherein capturing depth data with the depth camera at the first pose comprises capturing depth data using a stereo technique. 
     
     
         9 . The method of  claim 2 , wherein the depth camera is an assisted stereo depth camera system comprising two infrared cameras and a structured light projector; wherein capturing depth data with the depth camera at the first pose comprises capturing depth data using an assisted stereo technique. 
     
     
         10 . The method of  claim 2 , wherein capturing 2D image data at the first pose comprises capturing 2D image data at a first time; wherein capturing 2D image data at the second pose comprises capturing 2D image data after a threshold period of time has elapsed since the first time. 
     
     
         11 . The method of  claim 2 , wherein capturing 2D image data at the first pose comprises capturing 2D image data at a first time; wherein capturing 2D image data at the second pose comprises capturing 2D image data after:
 a threshold period of time has elapsed since the first time, and   motion has been detected by the 3D image capture system after the threshold period of time has elapsed.   
     
     
         12 . The method of  claim 2 , wherein identifying the set of points common to the depth data captured at the first pose and the depth data captured at the second pose comprises identifying the set of points using an iterative closest point algorithm. 
     
     
         13 . The method of  claim 2 , further comprising tracking motion of the 3D image capture system between the first and second poses; wherein identifying the set of points common to the depth data captured at the first pose and the depth data captured at the second pose comprises identifying the set of points by applying a rigid body transform based on the tracked motion. 
     
     
         14 . The method of  claim 2 , wherein calculating the color/texture error function comprises calculating a mean squared error function for the RGB values of each of the set of points. 
     
     
         15 . The method of  claim 14 , wherein updating registration data to reduce the color/texture error function comprises performing bounded iterative linearization. 
     
     
         16 . The method of  claim 2 , further comprising temporally synchronizing the RGB camera and the depth camera using timing signals. 
     
     
         17 . The method of  claim 16 , wherein temporally synchronizing the RGB camera and the depth camera comprises synchronizing the RGB camera to capture two frames for every frame captured by the depth camera. 
     
     
         18 . The method of  claim 2 , wherein performing the initial registration of the RGB camera to the depth camera comprises performing the initial registration using an image feature detection technique. 
     
     
         19 . The method of  claim 2 , wherein calculating the updated registration comprises calculating the updated registration according to a primary autoregistration technique; further comprising supplementing automatic registration by performing a secondary autoregistration technique; wherein the secondary autoregistration technique is distinct from the primary autoregistration technique. 
     
     
         20 . The method of  claim 19 , wherein supplementing automatic registration is performed in response to at least one of environmental conditions, capture system conditions, and performance conditions. 
     
     
         21 . The method of  claim 2 , further comprising supplementing automatic registration by receiving user input corresponding to one or more registration parameters and calculating a refined registration of the RGB camera to the depth camera based on the updated registration and the user input. 
     
     
         22 . The method of  claim 21 , wherein receiving user input comprises displaying a depth map image overlaid on an RGB image, enabling alignment of the depth map image and the RGB image, and measuring transformations performed by a user to align the depth map image and the RGB image. 
     
     
         23 . The method of  claim 22 , wherein enabling alignment of the depth map image and the RGB image comprises enabling translation of the depth map image relative to the RGB image and no other transformation. 
     
     
         24 . The method of  claim 22 , wherein calculating the refined registration comprises calculating the refined registration based on measured transformations performed by the user to align the depth map image and the RGB image. 
     
     
         25 . The method of  claim 2 , further comprising supplementing automatic registration by receiving user input corresponding to one or more registration parameters and calculating an intermediate registration of the RGB camera to the depth camera based on the user input; wherein calculating the updated registration of the RGB camera to the depth camera comprises calculating the updated registration based on the intermediate registration. 
     
     
         26 . The method of  claim 25 , wherein receiving user input comprises displaying a depth map image overlaid on an RGB image, enabling alignment of the depth map image and the RGB image, and measuring transformations performed by a user to align the depth map image and the RGB image. 
     
     
         27 . The method of  claim 1 , wherein capturing 2D image data with the RGB camera at the first pose comprises capturing an RGB image with the RGB camera, wherein capturing depth data with the depth camera at the first pose comprises capturing an infrared image with the depth camera, wherein calculating the updated registration comprises:
 identifying a set of points common to the RGB image and the infrared image;   calculating a transformation between the RGB image and the infrared image; and   calculating the updated registration based on the transformation.   
     
     
         28 . The method of  claim 27 , further comprising calculating the updated registration based on a second transformation, the second transformation describing a relationship between the infrared image and a depth map produced by the depth camera. 
     
     
         29 . The method of  claim 27 , wherein the depth camera is a stereo depth camera system comprising two infrared cameras; wherein capturing depth data with the depth camera at the first pose further comprises capturing depth data using a stereo technique. 
     
     
         30 . The method of  claim 27 , wherein the depth camera is an assisted stereo depth camera system comprising two infrared cameras and a structured light projector; wherein capturing depth data with the depth camera at the first pose further comprises capturing depth data using an assisted stereo technique.