Methods for automatic registration of 3d image data

Abstract

A method for automatic registration of 3D image data, captured by a 3D image capture system having an RGB camera and a depth camera, includes capturing 2D image data with the RGB camera at a first pose; capturing depth data with the depth camera at the first pose; performing an initial registration of the RGB camera to the depth camera; capturing 2D image data with the RGB camera at a second pose; capturing depth data at the second pose; and calculating an updated registration of the RGB camera to the depth camera.

1 - 30 . (canceled) 
     
     
         31 . A method comprising:
 capturing first 2D image data with a red-green-blue (RGB) camera of a 3D image capture system at a first time;   capturing first depth data with the depth camera of the 3D image capture system at the first time;   determining a first pose associated with the 3D image capture system at the first time;   performing registration of the RGB camera to the depth camera based at least in part on the first pose;   capturing second 2D image data with the RGB camera at a second time;   capturing second depth data at the second time;   determining a second pose associated with the 3D image capture system at the second time; and   correcting the registration of the RGB camera to the depth camera based at least in part on the second pose.   
     
     
         32 . The method of  claim 31 , wherein correcting the registration comprises:
 identifying a set of points common to the first depth-and the second depth data;   calculating a color/texture error function for the set of points;   updating the registration to reduce the color/texture error function; and   wherein identifying the set of points common to the first depth data and the second depth data comprises identifying the set of points using an iterative closest point algorithm.   
     
     
         33 . The method of  claim 31 , further comprising:
 identifying a set of points common to the first depth-and the second depth data;   calculating a color/texture error function for the set of points;   updating the registration to reduce the color/texture error function; and   tracking motion of the 3D image capture system between the first pose and the second pose; and   wherein identifying the set of points common to the first depth data and the second depth data comprises identifying the set of points by applying a rigid body transform based at least in part on the motion.   
     
     
         34 . The method of  claim 31 , further comprising:
 tracking motion of the 3D image capture system between the first pose and the second pose;   wherein capturing the second depth data comprises capturing depth data using a derived data technique;   wherein using the derived data technique comprises calculating the second depth data based at least in part on the first depth data.   
     
     
         35 . The method of  claim 31 , wherein the second time is occurs after a threshold period of time has elapsed since the first time. 
     
     
         36 . The method of  claim 31 , wherein updating the registration comprises performing bounded iterative linearization using, at least in part, the first 2D image data, the second 2D image data, the first depth data, the second depth data, the first pose, and the second pose. 
     
     
         37 . The method of  claim 31 , further comprising:
 temporally synchronizing the RGB camera and the depth camera based at least in part on timing signals provided to the RGB camera and the depth camera; and   causing the RGB camera to capture two frames for every frame captured by the depth camera.   
     
     
         38 . The method of  claim 31 , wherein capturing the second 2D image data with the RGB camera is in response to detecting a motion of a 3D image capture system. 
     
     
         39 . The method of  claim 31 , wherein performing the registration of the RGB camera to the depth camera comprises performing the registration using an image feature detection technique. 
     
     
         40 . The method of  claim 31 , further comprising:
 receiving user input corresponding to one or more registration parameters; and   correcting the registration of the RGB camera to the depth camera based on at least in part on the user input.   
     
     
         41 . The method of  claim 40 , wherein receiving user input comprises:
 displaying a depth map image overlaid on an RGB image on a display;   enabling user alignment of the depth map image and the RGB image on the display; and   measuring transformations resulting from the user alignment of the depth map image and the RGB image.   
     
     
         42 . A system comprising:
 a red-green-blue (RGB) camera for capturing two-dimensional (2D) image data associated with a physical environment;   a depth camera for capturing depth data associated with the physical environment;   an inertial measurement unit (IMU) for capturing motion data associated with the system;   one or more processors;   non-transitory computer-readable media storing computer-executable instructions, which when executed by the one or more processors cause the one or more processors to perform operations including:
 receiving first 2D image data captured by the red-green-blue (RGB) camera at a first time;
 receiving first depth data captured by the depth camera at the first time; 
 
 determining a first pose associated with the system at the first time based at least in part on the motion data at the first time; 
 performing registration of the RGB camera to the depth camera based at least in part on the first pose, the first 2D data, and the first depth data; 
 receiving second 2D image data captured by the RGB camera at a second time; 
 receiving second depth data captured by the depth camera at the second time; 
 determining a second pose associated with the system at the second time based at least in part on the motion data at the first time; and 
 correcting the registration of the RGB camera to the depth camera based at least in part on the second pose. 
   
     
     
         43 . The system of  claim 42 , wherein the second depth data and the second 2D image data is captured in response to a period of time elapsing. 
     
     
         44 . The system of  claim 42 , wherein the second depth data and the second 2D image data is captured in response to the motion data indicating a change in position of the system. 
     
     
         45 . The system of  claim 42 , wherein:
 the first 2D image data includes foreground of a first color against a background of a second color;   the second 2D image data includes foreground of the first color against a background of the second color;   the first depth data includes depth data associated with the foreground and the background of the first 2D image data;   the second depth data includes depth data associated with the foreground and the background of the second 2D image data; and   performing the registration includes:
 generating first combined image data by combining the first 2D image data with the first depth data; 
 detecting a profile of an object based at least in part on the combined image data; 
 generating second combined image data by combining the second 2D image data with the second depth data; 
 overlaying the second combined image data over the first combined image data; 
 detecting differences between the second combined image data and the first combined image data; 
 identify pixels of the first depth data corresponding to the differences; 
 identify pixels of the second depth data corresponding to the pixels of the first depth data; and 
 correcting the registration based at least in part on the pixels of the first depth data and the corresponding pixels of the second depth data. 
   
     
     
         46 . The system of  claim 42 , further comprising:
 wherein the non-transitory computer-readable media stores additional computer-executable instructions, which when executed by the one or more processors cause the one or more processors to perform operations including:   receiving user input corresponding to one or more registration parameters; and   correcting the registration of the RGB camera to the depth camera based on at least in part on the user input.   
     
     
         47 . A method comprising:
 receiving first 2D image data from a red-green-blue (RGB) camera of a three-dimensional (3D) system;   receiving first depth data from a depth camera, the first depth data and the first 2D image data captured at substantially a first time, the depth camera located on the 3D system at a location proximate to the RGB camera but different from the RGB camera;   determining a first pose associated with the 3D system at the first time;   performing registration of the RGB camera to the depth camera based at least in part on a comparison of the first 2D data to the first depth data;   receiving second 2D image data form the RGB camera;   receiving second depth data from the depth camera, the second depth data captured and the second 2D data captured at substantially a second time;   determining a second pose associated with the system at the second time; and   correcting the registration of the RGB camera to the depth camera based at least in part on the second pose, the second depth data, and the second 2D image data.   
     
     
         48 . The method of  claim 47 , further comprising correcting the registration of the RGB camera to the depth camera based on at least in part on a user input. 
     
     
         49 . The method of  claim 47 , wherein correcting the registration comprises:
 identifying a set of points common to the first depth-and the second depth data;   calculating a color/texture error function for the set of points; and   updating the registration to reduce the color/texture error function.   
     
     
         50 . The method of  claim 47 , wherein performing the registration of the RGB camera to the depth camera includes overlaying the depth data on the RGB image data.