Hologram streaming machine

Abstract

Example systems and methods perform streaming of volumetric media and accommodate high user interactivity. A device is configured to access and render streaming holograms and may implement a window as a buffer. In addition, a hologram streaming machine can be configured to stream full or partial holograms in the form of 3D blocks, where different 3D blocks represent a same portion of hologram but may have different resolutions depending on where the user is positioned and looking relative to each 3D block, thus saving network capacity by focusing on what the user is looking at. Since many 3D blocks may be empty much of the time, may be occluded or far away from the user's viewing position, or may be numerous within a 3D space, the device can be configured to request 3D blocks based on their utility, which may be calculated based on bitrate, visibility, or distance.

What is claimed is: 
     
         1 . A method comprising:
 detecting, by one or more processors, that a first view of a three-dimensional (3D) object is to be replaced by a second view of the 3D object, the first view corresponding to a first set of 3D blocks stored in a buffer, the first set of 3D blocks partitioning at least a first portion of the 3D object and each having a different combination of presentation time and presentation location;   selecting, by the one or more processors, a second set of 3D blocks that partition at least a second portion of the 3D object and that each have a different combination of presentation time and presentation location, the selected second set including a second 3D block;   inserting, by the one or more processors, the selected second 3D block into the buffer that already stores the first set of 3D blocks; and   causing, by the one or more processors, the second 3D block to be rendered in the second view of the 3D object.   
     
     
         2 . The method of  claim 1 , wherein:
 a first 3D block in the first set of 3D blocks has a first presentation time, and the second 3D block has a second presentation time earlier than the first presentation time of the first 3D block; and   the second 3D block is non-sequentially inserted into the buffer in response to the second presentation time of the second 3D block being earlier than the first presentation time of the first 3D block.   
     
     
         3 . The method of  claim 1 , wherein:
 the first view depicts the first portion of the 3D object in a first viewing frustum within a 3D space;   the second view depicts the second portion of the 3D object in a second viewing frustum within the 3D space; and   the first and second viewing frustums within the 3D space differ in at least one of viewing direction or viewing position.   
     
     
         4 . The method of  claim 1 , wherein:
 the 3D object includes a 3D point cloud enclosed by a 3D bounding volume;   the first set of 3D blocks occupies at least a first portion of the 3D bounding volume, the first set of 3D blocks being renderable during a first time period that includes their corresponding presentation times and within a first viewing frustum that includes their corresponding presentation locations; and   the second set of 3D blocks occupies at least a second portion of the 3D bounding volume, the second set of 3D blocks being renderable during a second time period that includes their corresponding presentation times and within a second viewing frustum that includes their corresponding presentation locations.   
     
     
         5 . The method of  claim 1 , wherein:
 the selecting of the second set of 3D blocks includes:
 accessing manifest data that describes representations of the 3D object, the representations differing in visual quality; 
 selecting a representation of the 3D object among the representations of the 3D object, the manifest data correlating the selected representation of the 3D object to the second 3D block; and 
 selecting the second 3D block based on its correlation with the selected representation of the 3D object. 
   
     
     
         6 . The method of  claim 5 , wherein:
 the representations of the 3D object differ from each other in at least one of bitrates, spatial resolutions, or temporal resolutions; and   the selecting of the representation of the 3D object is based on at least one of a bitrate of the representation, a spatial resolution of the representation, or a temporal resolution of the representation.   
     
     
         7 . The method of  claim 1 , wherein:
 the selecting of the second set of 3D blocks includes:
 accessing an index that indicates a presentation location of the second 3D block; and 
 selecting the second 3D block based on its presentation location. 
   
     
     
         8 . The method of  claim 1 , wherein:
 the selecting of the second set of 3D blocks includes:
 accessing an index that indicates a normal vector of the second 3D block; and 
 selecting the second 3D block based on its normal vector. 
   
     
     
         9 . The method of  claim 1 , wherein:
 the selecting of the second set of 3D blocks includes:
 calculating a utility score of the second 3D block; and 
 selecting the second 3D block based on its calculated utility score. 
   
     
     
         10 . The method of  claim 1 , wherein:
 the selecting of the second set of 3D blocks includes:
 calculating a visibility score of the second 3D block; and 
 selecting the second 3D block based on its calculated visibility score. 
   
     
     
         11 . A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
 detecting that a first view of a three-dimensional (3D) object is to be replaced by a second view of the 3D object, the first view corresponding to a first set of 3D blocks stored in a buffer, the first set of 3D blocks partitioning at least a first portion of the 3D object and each having a different combination of presentation time and presentation location;   selecting a second set of 3D blocks that partition at least a second portion of the 3D object and that each have a different combination of presentation time and presentation location, the selected second set including a second 3D block;   inserting the selected second 3D block into the buffer that already stores the first set of 3D blocks; and   causing the second 3D block to be rendered in the second view of the 3D object.   
     
     
         12 . The non-transitory machine-readable storage medium of  claim 11 , wherein:
 a first 3D block in the first set of 3D blocks has a first presentation time, and the second 3D block has a second presentation time earlier than the first presentation time of the first 3D block; and   the second 3D block is non-sequentially inserted into the buffer in response to the second presentation time of the second 3D block being earlier than the first presentation time of the first 3D block.   
     
     
         13 . A system comprising:
 one or more processors; and   a memory storing instructions that, when executed by at least one processor among the one or more processors, cause the system to perform operations comprising:   detecting that a first view of a three-dimensional (3D) object is to be replaced by a second view of the 3D object, the first view corresponding to a first set of 3D blocks stored in a buffer, the first set of 3D blocks partitioning at least a first portion of the 3D object and each having a different combination of presentation time and presentation location;   selecting a second set of 3D blocks that partition at least a second portion of the 3D object and that each have a different combination of presentation time and presentation location, the selected second set including a second 3D block;   inserting the selected second 3D block into the buffer that already stores the first set of 3D blocks; and   causing the second 3D block to be rendered in the second view of the 3D object.   
     
     
         14 . The system of  claim 13 , wherein:
 the selecting of the second set of 3D blocks includes:
 accessing manifest data that describes representations of the 3D object, the representations differing in visual quality; 
 selecting a representation of the 3D object among the representations of the 3D object, the manifest data correlating the selected representation of the 3D object to the second 3D block; and 
 selecting the second 3D block based on its correlation with the selected representation of the 3D object. 
   
     
     
         15 . A method comprising:
 providing, by one or more processors, a first set of three-dimensional (3D) blocks that correspond to a first view of a 3D object, that partition at least a first portion of the 3D object, and that each have a different combination of presentation time and presentation location;   detecting, by the one or more processors, a request for a second 3D block, the request resulting from selection of a second set of 3D blocks based on a detection that a second view of the 3D object is to replace the first view of the 3D object, the selected second set of 3D blocks partitioning at least a second portion of the 3D object and each having a different combination of presentation time and presentation location; and   providing, in response to the request and by the one or more processors, the requested second 3D block, the provided second 3D block being rendered in the second view of the 3D object.   
     
     
         16 . The method of  claim 11 , wherein:
 the first view depicts the first portion of the 3D object in a first viewing frustum within a 3D space;   the second view depicts the second portion of the 3D object in a second viewing frustum within the 3D space; and   the first and second viewing frustums within the 3D space differ in at least one of viewing direction or viewing position.   
     
     
         17 . A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
 providing a first set of three-dimensional (3D) blocks that correspond to a first view of a 3D object, that partition at least a first portion of the 3D object, and that each have a different combination of presentation time and presentation location;   detecting a request for a second 3D block, the request resulting from selection of a second set of 3D blocks based on a detection that a second view of the 3D object is to replace the first view of the 3D object, the selected second set of 3D blocks partitioning at least a second portion of the 3D object and each having a different combination of presentation time and presentation location; and   providing, in response to the request, the requested second 3D block, the provided second 3D block being rendered in the second view of the 3D object.   
     
     
         18 . The non-transitory machine-readable storage medium of  claim 17 , wherein:
 the 3D object includes a 3D point cloud enclosed by a 3D bounding volume;   the first set of 3D blocks occupies at least a first portion of the 3D bounding volume, the first set of 3D blocks being renderable during a first time period that includes their corresponding presentation times and within a first viewing frustum that includes their corresponding presentation locations; and   the second set of 3D blocks occupies at least a second portion of the 3D bounding volume, the second set of 3D blocks being renderable during a second time period that includes their corresponding presentation times and within a second viewing frustum that includes their corresponding presentation locations.   
     
     
         19 . A system comprising:
 one or more processors; and   a memory storing instructions that, when executed by at least one processor among the one or more processors, cause the system to perform operations comprising:   providing a first set of three-dimensional (3D) blocks that correspond to a first view of a 3D object, that partition at least a first portion of the 3D object, and that each have a different combination of presentation time and presentation location;   detecting a request for a second 3D block, the request resulting from selection of a second set of 3D blocks based on a detection that a second view of the 3D object is to replace the first view of the 3D object, the selected second set of 3D blocks partitioning at least a second portion of the 3D object and each having a different combination of presentation time and presentation location; and   providing, in response to the request, the requested second 3D block, the provided second 3D block being rendered in the second view of the 3D object.   
     
     
         20 . The system of  claim 19 , wherein:
 the first view depicts the first portion of the 3D object in a first viewing frustum within a 3D space;   the second view depicts the second portion of the 3D object in a second viewing frustum within the 3D space; and   the first and second viewing frustums within the 3D space differ in at least one of viewing direction or viewing position.