Generation, transmission and rendering of virtual reality multimedia

Abstract

A method of generating virtual reality data includes: obtaining point cloud data, the point cloud data including colour and three-dimensional position data for each of a plurality of points corresponding to locations in a capture volume; generating primary image data containing (i) a first projection of a first subset of the points into a two-dimensional frame of reference, and (ii) for each point of the first subset, depth data derived from the corresponding position data; generating secondary image data containing (i) a second projection of a second subset of the points into the two-dimensional frame of reference, the second projection overlapping with at least part of the first projection in the two-dimensional frame of reference, and (ii) for each point of the second subset, depth data derived from the corresponding position data; and storing the primary image data and the secondary image data in a memory.

1 . A method of generating virtual reality multimedia data, comprising:
 obtaining point cloud data at a processor of a generation computing device, the point cloud data including colour and three-dimensional position data for each of a plurality of points corresponding to locations in a capture volume;   generating, at the processor, primary image data containing (i) a first projection of a first subset of the points into a two-dimensional frame of reference, and (ii) for each point of the first subset, depth data derived from the corresponding position data;   generating, at the processor, secondary image data containing (i) a second projection of a second subset of the points into the two-dimensional frame of reference, the second projection overlapping with at least part of the first projection in the two-dimensional frame of reference, and (ii) for each point of the second subset, depth data derived from the corresponding position data; and   storing the primary image data and the secondary image data in a memory connected to the processor.   
     
     
         2 . The method of  claim 1 , wherein obtaining the point cloud data includes retrieving the point cloud data from a memory. 
     
     
         3 . The method of  claim 1 , wherein obtaining the point cloud data includes:
 receiving raw point cloud data from a capture apparatus; and   generating the point cloud data by registering the raw point cloud data to a common three-dimensional frame of reference.   
     
     
         4 . The method of  claim 1 , the primary image data including:
 a first image dimensioned according to the two-dimensional frame of reference and containing colour data for each point of the first subset; and   a second image dimensioned according to the two-dimensional frame of reference and containing depth data for each point of the first subset.   
     
     
         5 . The method of  claim 4 , wherein the first image and the second image are cube map projections. 
     
     
         6 . The method of  claim 4 , wherein the first image and the second image are YUV images having luminance and chrominance channels, and wherein the depth data includes a depth value stored in the luminance channel. 
     
     
         7 . The method of  claim 1 , the secondary image data including:
 a first image dimensioned according to the two-dimensional frame of reference and containing colour data for each point of the second subset; and   a second image dimensioned according to the two-dimensional frame of reference and containing depth data for each point of the first subset.   
     
     
         8 . The method of  claim 1 , wherein the first image and the second image are cube map projections. 
     
     
         9 . The method of  claim 7 , further comprising:
 detecting that a plurality of colliding ones of the second subset of points have a common position in the two-dimensional frame of reference;   storing colour data and depth data for one of the colliding points in the first image and the second image according to the common position;   storing colour data and depth data for another of the colliding points in the first image and the second image at a two-dimensional offset from the common position.   
     
     
         10 . The method of  claim 9 , further comprising:
 storing the two-dimensional offset in the second image.   
     
     
         11 . The method of  claim 1 , wherein generating the primary image data includes:
 setting a viewpoint position corresponding to a location in the capture volume; and   selecting the first subset of the points by:
 for each of a plurality of paths extending from the viewpoint position, selecting the first point of the point cloud data encountered by the path. 
   
     
     
         12 . The method of  claim 1 , wherein generating the primary image data includes:
 setting a viewpoint position corresponding to a location in the capture volume; and   selecting the first subset of the points by:
 determining a distance from the viewpoint to each of the plurality of points; 
 comparing the distance to a threshold; and 
 selecting the points having a smaller distance than the threshold from the viewpoint. 
   
     
     
         13 . The method of  claim 1 , further comprising:
 transmitting the primary image data and the secondary image data for receipt by a client device.   
     
     
         14 . A generation computing device, comprising:
 a memory;   a network interface; and   a processor interconnected with the memory and the network interface, the processor configured to:
 obtain point cloud data at a processor of a generation computing device, the point cloud data including colour and three-dimensional position data for each of a plurality of points corresponding to locations in a capture volume; 
 generate primary image data containing (i) a first projection of a first subset of the points into a two-dimensional frame of reference, and (ii) for each point of the first subset, depth data derived from the corresponding position data; 
 generate secondary image data containing (i) a second projection of a second subset of the points into the two-dimensional frame of reference, the second projection overlapping with at least part of the first projection in the two-dimensional frame of reference, and (ii) for each point of the second subset, depth data derived from the corresponding position data; and 
 store the primary image data and the secondary image data in the memory. 
   
     
     
         15 . A method of rendering virtual reality multimedia data, comprising:
 obtaining primary image data containing (i) a first projection of a first subset of points in a three-dimensional point cloud into a two-dimensional frame of reference, and (ii) for each point of the first subset, depth data derived from corresponding position data of the points;   obtaining secondary image data containing (i) a second projection of a second subset of the points into the two-dimensional frame of reference, the second projection overlapping with at least part of the first projection in the two-dimensional frame of reference, and (ii) for each point of the second subset, depth data derived from the corresponding position data;   receiving a viewpoint position from a virtual reality display;   selecting at least a portion of the primary image data and the secondary image data based on the viewpoint position; and   rendering the selected primary and secondary image data on the virtual reality display.   
     
     
         16 . A client computing device, comprising:
 a memory;   a network interface; and   a processor interconnected with the memory and the network interface, the processor configured to:
 obtain primary image data containing (i) a first projection of a first subset of points in a three-dimensional point cloud into a two-dimensional frame of reference, and (ii) for each point of the first subset, depth data derived from corresponding position data of the points; 
 obtain secondary image data containing (i) a second projection of a second subset of the points into the two-dimensional frame of reference, the second projection overlapping with at least part of the first projection in the two-dimensional frame of reference, and (ii) for each point of the second subset, depth data derived from the corresponding position data; 
 receive a viewpoint position from a virtual reality display; 
 select at least a portion of the primary image data and the secondary image data based on the viewpoint position; and 
 render the selected primary and secondary image data on the virtual reality display.