Three-dimensional scene reconstruction from set of two dimensional images for consumption in virtual reality

Abstract

To enable better sharing and preservation of immersive experiences, a graphics system reconstructs a three-dimensional scene from a set of images of the scene taken from different vantage points. The system processes each image to extract depth information therefrom and then stitches the images (both color and depth information) into a multi-layered panorama that includes at least front and back surface layers. The front and back surface layers are then merged to remove redundancies and create connections between neighboring pixels that are likely to represent the same object, while removing connections between neighboring pixels that are not. The resulting layered panorama with depth information can be rendered using a virtual reality (VR) system, a mobile device, or other computing and display platforms using standard rendering techniques, to enable three-dimensional viewing of the scene.

What is claimed is: 
     
         1 . A method comprising:
 receiving a plurality of input images of a scene taken from different vantage points;   processing the plurality of input images to generate a representation including a sparse point cloud specifying locations of a plurality of points corresponding to three-dimensional locations of surfaces of objects in the scene;   generating depth images for the plurality of input images based on the representation including the sparse point cloud;   generating, from the depth images, a two-layer panorama including a front surface panorama and a back surface panorama; and   generating, from the two-layer panorama, a three-dimensional image suitable for rendering the scene in the three-dimensional space.   
     
     
         2 . The method of  claim 1 , wherein generating the two-layer panorama comprises:
 projecting front surfaces of the depth images using a forward depth test to generate a plurality of front-warped images;   projecting back surfaces of the depth images using an inverted depth test to generate a plurality of back-warped images; and   stitching the front-warped images and the back-warped images to generate the two-layer panorama.   
     
     
         3 . The method of  claim 2 , wherein stitching the front-warped images and the back-warped images to generate a two-layer panorama comprises:
 stitching a depth panorama using depth values from the front-warped images;   stitching the front surface panorama using color values from the front-warped images and stitched depth values from the depth panorama;   stitching the back surface panorama using color values from the back-warped images and the stitched depth values from the depth panorama; and   combining the front surface panorama and the back surface panorama into the two-layer panorama.   
     
     
         4 . The method of  claim 1 , wherein generating the three-dimensional image comprises:
 fusing a front surface panorama and a back surface panorama of the two-layer panorama to generate the three-dimensional image.   
     
     
         5 . The method of  claim 1 , wherein fusing the front surface panorama and the back surface panorama comprises:
 removing background pixels from the back surface panorama that match corresponding foreground pixels in the front surface panorama;   storing connections between neighboring pixels meeting a threshold similarity in depth and color information; and   hallucinating color and depth information in missing pixel locations.   
     
     
         6 . The method of  claim 1 , wherein processing the plurality of input images to generate the sparse reconstruction representation comprises:
 applying a surface-from-motion algorithm to the plurality of input images.   
     
     
         7 . The method of  claim 1 , wherein processing the plurality of input images to generate the representation including the sparse point cloud comprises:
 generating a near envelope prior that assigns a cost to estimated depth values in front of a near envelope; and   applying a multi-view stereo processing algorithm to estimate the depth values based on a cost function including the near envelope prior.   
     
     
         8 . The method of  claim 7 , wherein generating the near envelope prior comprises:
 identifying anchor pixels in the plurality of input images that have high confidence depth estimates;   propagating the depth estimates of the anchor pixels to other pixels in the plurality of input images to generate approximate depth maps;   filtering the approximate depth maps to determine a near envelope; and   generating the near envelope prior based on the depth estimates and the near envelope.   
     
     
         9 . The method of  claim 1 , further comprising:
 generating a normal map associated with the three-dimensional image, the normal map estimating for each pixel, an angle normal to a surface depicted by the pixel.   
     
     
         10 . The method of  claim 9 , wherein generating the normal map comprises:
 generating a base normal map from depth values in the three-dimensional image;   generating a detailed normal from luminance values in the three-dimensional image; and   transforming the detailed normal map onto the base normal map to generate a combined normal map.   
     
     
         11 . The method of  claim 1 , wherein the plurality of input images having varying levels of overlap and orientation changes. 
     
     
         12 . A non-transitory computer-readable storage medium storing instructions, the instructions when executed by a processor causing the processor to perform steps including:
 receiving a plurality of input images of a scene taken from different vantage points;   processing the plurality of input images to generate a representation including a sparse point cloud specifying locations of a plurality of points corresponding to three-dimensional locations of surfaces of objects in the scene;   generating depth images for the plurality of input images based on the representation including the sparse point cloud;   generating, from the depth images, a two-layer panorama including a front surface panorama and a back surface panorama; and   generating, from the two-layer panorama, a three-dimensional image suitable for rendering the scene in the three-dimensional space.   
     
     
         13 . The non-transitory computer-readable storage medium of  claim 12 , wherein generating the two-layer panorama comprises:
 projecting front surfaces of the depth images using a forward depth test to generate a plurality of front-warped images;   projecting back surfaces of the depth images using an inverted depth test to generate a plurality of back-warped images; and   stitching the front-warped images and the back-warped images to generate the two-layer panorama.   
     
     
         14 . The non-transitory computer-readable storage medium of  claim 12 , wherein generating the three-dimensional image comprises:
 fusing a front surface panorama and a back surface panorama of the two-layer panorama to generate the three-dimensional image.   
     
     
         15 . The non-transitory computer-readable storage medium of  claim 12 , wherein processing the plurality of input images to generate the sparse reconstruction representation comprises:
 applying a surface-from-motion algorithm to the plurality of input images.   
     
     
         16 . The method of  claim 1 , wherein processing the plurality of input images to generate the representation including the sparse point cloud comprises:
 generating a near envelope prior that assigns a cost to estimated depth values in front of a near envelope; and   applying a multi-view stereo processing algorithm to estimate the depth values based on a cost function including the near envelope prior.   
     
     
         17 . The non-transitory computer-readable storage medium of  claim 12 , the instructions when executed further causing the processor to perform steps including:
 generating a normal map associated with the three-dimensional image, the normal map estimating for each pixel, an angle normal to a surface depicted by the pixel.   
     
     
         18 . The non-transitory computer-readable storage medium of  claim 12 , wherein the plurality of input images having varying levels of overlap and orientation changes. 
     
     
         19 . A system comprising:
 a processor; and   a non-transitory computer-readable storage medium storing instruction for generating a three-dimensional image, the instructions when executed by a processor causing the processor to perform steps including:
 receiving a plurality of input images of a scene taken from different vantage points; 
 processing the plurality of input images to generate a representation including a sparse point cloud specifying locations of a plurality of points corresponding to three-dimensional locations of surfaces of objects in the scene; 
 generating depth images for the plurality of input images based on the representation including the sparse point cloud; 
 generating, from the depth images, a two-layer panorama including a front surface panorama and a back surface panorama; and 
 generating, from the two-layer panorama, a three-dimensional image suitable for rendering the scene in the three-dimensional space. 
   
     
     
         20 . The system of  claim 19 , the instructions when executed further causing the processor to perform steps including:
 generating a normal map associated with the three-dimensional image, the normal map estimating for each pixel, an angle normal to a surface depicted by the pixel.