Layered scene decomposition codec with higher order lighting

Abstract

A system and methods for a CODEC driving a real-time light field display for multi-dimensional video streaming, interactive gaming and other light field display applications is provided applying a layered scene decomposition strategy. Multi-dimensional scene data including information on directions of normals is divided into a plurality of data layers of increasing depths as the distance between a given layer and the display surface increases. Data layers which are sampled using a plenoptic sampling scheme and rendered using hybrid rendering, such as perspective and oblique rendering, to encode light fields corresponding to each data layer. The resulting compressed, (layered) core representation of the multi-dimensional scene data is produced at predictable rates, reconstructed and merged at the light field display in real-time by applying view synthesis protocols, including edge adaptive interpolation, to reconstruct pixel arrays in stages (e.g. columns then rows) from reference elemental images.

1 . A computer-implemented method comprising:
 receiving a first data set comprising a three-dimensional description of a scene, the first data set comprising information on directions of normals on surfaces in the scene with respect to a reference direction;   partitioning the first data set into a plurality of layers, each layer representing a portion of the scene at a location with respect to a reference location;   encoding the plurality of layers to generate a second data set, wherein the second data set is smaller in size than the first data set;   calculating specular lighting by reconstructing the plurality of layers using the directions of normals on surfaces included in the scene to provide constructed layered light fields;   merging the constructed layered light fields to produce a final light field image; and   displaying the three-dimensional scene.   
     
     
         2 . The method of  claim 1 , wherein calculating the specular lighting by reconstructing the plurality of layers is done using a multi-stage view synthesis reconstruction. 
     
     
         3 . The method of  claim 2 , wherein the view synthesis reconstruction is done using a warping process, screen space ray tracing, machine learning, image-based rendering, or a combination thereof. 
     
     
         4 . The method of  claim 1 , wherein at least some of the surfaces in the scene have non-Lambertian reflection properties. 
     
     
         5 . The method of  claim 1 , wherein the constructed layered light fields represents inner frustrum and outer frustum volumes of the final light field image. 
     
     
         6 . The method of  claim 1 , wherein the surface normal information relative to a light position, or point, may be known and included with encoded light field data. 
     
     
         7 . The method of  claim 1 , further comprising storing the normals on surfaces in combination with RGB and depth information. 
     
     
         8 . The method of  claim 1 , wherein the method captures at least one of gloss, haze, and goniochromatic color of an object in the scene. 
     
     
         9 . The method of  claim 1 , further comprising applying shading as a post-process to reconstructed pixels in the constructed layered light fields. 
     
     
         10 . The method of  claim 1 , wherein encoding the plurality of layers comprises performing a sampling operation on at least a portion of the first data set to generate the second data set. 
     
     
         11 . The method of  claim 10 , wherein performing the sampling operation is based on a target compression rate associated with the second data set. 
     
     
         12 . The method of  claim 10 , wherein performing the sampling operation comprises selecting multiple elemental images from a plurality of elemental images in accordance with a plenoptic sampling scheme. 
     
     
         13 . The method of  claim 10 , wherein performing the sampling operation comprises:
 determining an effective spatial resolution associated with each layer; and   selecting multiple elemental images from a plurality of elemental images in accordance with a determined angular resolution.   
     
     
         14 . The method of  claim 13 , wherein the angular resolution is determined as a function of a directional resolution associated with the portion of the scene associated with each layer. 
     
     
         15 . The method of  claim 13 , wherein the angular resolution is determined as a field of view associated with a display device. 
     
     
         16 . The method of  claim 1 , wherein encoding the plurality of layers comprises:
 rendering using ray tracing, a set of pixels to be encoded;   selecting multiple elemental images from a plurality of elemental images such that the set of pixels are rendered using the selected multiple elemental images; and   sampling the set of pixels using a sampling operation.   
     
     
         17 . The method of  claim 1 , wherein the three-dimensional description comprises light field data representing a plurality of elemental images. 
     
     
         18 . The method of  claim 17 , wherein the light field data includes a depth map corresponding to the elemental images. 
     
     
         19 . The method of  claim 17 , wherein each of the plurality of elemental images is captured by one or more image acquisition devices. 
     
     
         20 . The method of  claim 1 , further comprising:
 receiving user-input indicative of a location of a user with respect to the final light field; and   updating the final light field in accordance with the user-input prior to displaying the three-dimensional scene.   
     
     
         21 . The method of  claim 1 , wherein the information on the directions of normals is stored in a geometry buffer. 
     
     
         22 . The method of  claim 21 , wherein the geometry buffer stores color and depth information. 
     
     
         23 . The method of  claim 1 , wherein partitioning the first data set into a plurality of layers comprises restricting a depth range of each layer. 
     
     
         24 . The method of  claim 1 , wherein layers in the plurality of layers located closer to the display surface are narrower in width than layers located farther away from the display surface. 
     
     
         25 . The method of  claim 1 , wherein partitioning the first data set into a plurality of layers maintains a uniform compression rate across the scene. 
     
     
         26 . The method of  claim 1 , wherein the method is used to generate a synthetic light field for multi-dimensional video streaming, multi-dimensional interactive gaming, or real-time interactive content. 
     
     
         27 . The method of  claim 26 , wherein the synthetic light field is generated only in a valid viewing zone.