System for Generating a Three-Dimensional Scene Reconstructions

Abstract

A system configured to generate a three-dimensional scene reconstruction of a physical environment. In some cases, the system may store the three-dimensional scene reconstruction as two or more meshes and/or as one or more ray bundles including a plurality of depth values from a center point of the bundle.

What is claimed is: 
     
         1 . A device comprising:
 a display;   one or more image components for capturing image data associated with a physical environment surrounding a user;   one or more processors; and   non-transitory computer-readable media storing computer-executable instructions, which when executed by the one or more processors cause the one or more processors to perform operations comprising:
 receiving plurality of frames associated with the physical environment from the one or more image components; 
 selecting a center point associated with the plurality of frames; 
 casting a ray from the center point; 
 estimating a first position and a second position along the ray; 
 projecting the first position into the plurality of frames to determine a first consistency value; 
 projecting the second position into the plurality of frames to determine a second consistency value; 
 determining a depth value associated with the ray based at least in part on the first consistency value and the second consistency value; and 
 generating a three-dimensional scene reconstruction based at least in part on the depth value. 
   
     
     
         2 . The device as recited in  claim 1 , further comprising one or more position sensors for capturing orientation and motion data associated with the device and wherein generating the three-dimensional scene reconstruction is based at least in part on the orientation and motion data. 
     
     
         3 . The device as recited in  claim 1 , wherein:
 the center point is a first center point;   generating the three-dimensional scene reconstruction further comprises generating a first viewpoint of the three-dimensional scene reconstruction, the first viewpoint associated with the first center point; and   wherein the operations further comprise:
 selecting a second center point, the second center point within a threshold distance of the first center point; 
 casting a second ray from the second center point; 
 estimating a first position and a second position along the second ray; 
 projecting the first position of the second ray into the plurality of frames to determine a third consistency value; 
 projecting the second position of the second ray into the plurality of frames to determine a fourth consistency value; 
   determining a depth value associated with the second ray based at least in part on the third consistency value and the fourth consistency value; and   generating a second viewpoint of the three-dimensional scene reconstruction and associated with the second center point based at least in part on the second depth value.   
     
     
         4 . The device as recited in  claim 1 , wherein the operations further comprise:
 defining a first region within the plurality of frames;   defining a second region within the plurality of frames;   determining the first region has a second depth value greater than a third depth value associated with the second region;   assigning, at least in part in response to determining the second depth value is greater than the third depth, the first region to a background mesh; and   assigning, at least in part in response to determining the second depth value is greater than the third depth, the second region to a foreground mesh.   
     
     
         5 . The device as recited in  claim 4 , wherein the operations further comprise defining a boundary between the first region and the second region based at last in part on a discontinuity in a depth associated with the first region and the second region. 
     
     
         6 . The device as recited in  claim 1 , wherein the operations further comprise determining a boundary of a navigable region of the three-dimensional scene reconstruction based at least in part on the center point. 
     
     
         7 . The device as recited in  claim 1 , wherein the ray is a first ray, the depth value is a first depth value, and the operations further comprising:
 casting a second ray from the center point, the second ray extending sharing a trajectory with the first ray;   estimating a third position and a fourth position along the second ray;   projecting the third position into the plurality of frames to determine a third consistency value;   projecting the fourth position into the plurality of frames to determine a fourth consistency value; and   determining a second depth value associated with the second ray based at least in part on the third consistency value and the fourth consistency value, the second depth value different than the first depth value.   
     
     
         8 . The device as recited in  claim 1 , wherein the first consistency value includes associated with one or more of the following:
 a color consistency;   a texture consistency;   a normal consistency;   a gradient consistency; or   a feature descriptor consistency.   
     
     
         9 . The device as recited in  claim 1 , wherein the depth value is a first depth value and the operations further comprising:
 estimating a third position and a fourth position along the ray;   projecting the third position into the plurality of frames to determine a third consistency value;   projecting the fourth position into the plurality of frames to determine a fourth consistency value; and   determining a second depth value associated with the ray based at least in part on the third consistency value and the fourth consistency value, the second depth value different than the first depth value.   
     
     
         10 . The device as recited in  claim 1 , wherein:
 determining the depth value for a first individual projection further comprise applying additional constraints; and   the additional constraint includes one or more of the following:
 a Manhattan world constraint; 
 a vertical constraint; 
 a habitational constraint; 
 a perpendicular; 
 a parallel; 
 a normal constraint; 
 a smoothness constraint; 
 a planarity constraint; or 
 a semantic-based constraint. 
   
     
     
         11 . The device as recited in  claim 1 , wherein the operations further comprise segmenting and classifying data associated with the plurality of frames. 
     
     
         12 . A method comprising:
 determining a point associated with image data representing a three-dimensional physical environment;   determining an estimated depth associated with a ray cast from the point;   projecting the estimated depth into a first frame of the image data to determine a first value;   projecting the estimated depth into a second frame of the image data to determine a second value;   determining a final depth value of the ray based at least in part on the first value and the second value; and   generating a three-dimensional scene reconstruction of the physical environment based at least in part on the final depth.   
     
     
         13 . The method as recited in  claim 12 , wherein the point includes multiple associated rays, a position of individual rays are at predetermined interval from the at least one other ray. 
     
     
         14 . The method as recited in  claim 12 , wherein determining the final depth value of the ray based at least in part on a visual consistency value associated with the first value and the second value. 
     
     
         15 . The method as recited in  claim 12 , further comprising:
 determining the first surface depth is associated with a first surface and a second surface depth is associated with a second surface;   determining a value representative of a difference between the first surface depth value and the second surface depth; and   assigning a first surface to a first mesh and a second surface to a second mesh based at least in part on the value.   
     
     
         16 . The method as recited in  claim 15 , further comprising:
 filling holes associated with the first mesh independently from operations associated with the second mesh.   
     
     
         17 . One or more non-transitory computer-readable media storing computer-executable instructions, which when executed by one or more processors cause the one or more processors to perform operations comprising:
 receiving a plurality of frames representative of a physical environment;   selecting a point, the point related to the plurality of frames;   generating a ray, for a plurality of degrees of the point with respect to the point;   determining a depth value for individual rays; and   storing the depth values as a ray bundle associated with the physical environment.   
     
     
         18 . The device as recited in  claim 17 , further comprising generating a three-dimensional scene reconstruction based at least in part on the ray bundle. 
     
     
         19 . The device as recited in  claim 17 , wherein determining the depth value for a first individual ray is based at least in part on a normal associated with a region of an individual frame of the plurality of frames, the individual frame associated with the first individual ray. 
     
     
         20 . The device as recited in  claim 17 , further comprising defining a boundary between a first surface and a second surface based at least in part on a discontinuity in a depth associated with pixels of the first surface and the second surface.