Encoding and decoding of volumetric video

Abstract

There are disclosed various apparatuses, methods, and computer program products for coding volumetric video, where information of a three-dimensional scene is obtained and at least one projection plane for the scene is determined (602). At least a first projection direction is determined for a first part of the projection plane and a second projection direction is determined for a second part of the projection plane so that the first projection direction is different from the second projection direction (604). A first part of the scene is projected on the projection plane from the first projection direction (610); and a second part of the scene is projected on the projection plane from the second projection direction (610). The projection plane is encoded (612). Information of at least one projection plane of a three-dimensional scene, information of at least a first projection direction for a first part of the projection plane and a second projection direction for a second part of the projection plane are decoded. The first projection direction is different from the second projection direction. The decoded information is used to obtain reconstructed geometry data of different parts of the projection plane, and the reconstructed geometry data is used to form a reconstructed 3D image.

The invention claimed is: 
     
       1. An apparatus comprising at least one processor, non-transitory memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
 obtain information of a three-dimensional scene; 
 determine a two-dimensional projection plane for the scene; 
 determine a first concentration of at least one first projection direction for a first part of the two-dimensional projection plane, based on a determined first quality level to project a first part of the scene; 
 determine a second concentration of at least one second projection direction for a second part of the two-dimensional projection plane, based on a determined second quality level to project a second part of the scene; 
 wherein the first concentration of the at least one first projection direction is higher than the second concentration of the at least one second projection direction, in response to the first quality level being higher than the second quality level; 
 wherein the first projection direction for the first part of the two-dimensional projection plane is different from the second projection direction for the second part of the two-dimensional projection plane; 
 project a first part of the scene on the two-dimensional projection plane from the first projection direction; 
 project a second part of the scene on the two-dimensional projection plane from the second projection direction; and 
 encode the two-dimensional projection plane. 
 
     
     
       2. The apparatus according to  claim 1 , said memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
 encode information of the at least one first projection direction and the at least one second projection direction; and 
 signal the information of the at least one first projection direction and the at least one second projection direction to a decoder; 
 wherein the information encoded and signaled comprises the first concentration of at least one first projection direction, and the second concentration of the at least one second projection direction. 
 
     
     
       3. The apparatus according to  claim 2 , said memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
 encode said information of the first projection direction as a difference between the first projection direction and a reference direction; 
 wherein the reference direction comprises a surface normal of the two-dimensional projection plane toward a source volume within the scene. 
 
     
     
       4. The apparatus according to  claim 3 , said memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to encode said information of the second projection direction as one of the following:
 a difference between the second projection direction and the reference direction; or 
 a difference between the second projection direction and the first projection direction. 
 
     
     
       5. The apparatus according to  claim 1 , said memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
 determine projection directions for the two-dimensional projection plane in X- and Y-directions. 
 
     
     
       6. The apparatus according to  claim 1 , said memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
 determine the first projection direction and the second projection direction so that the first part and the second part of the two-dimensional projection plane differ from each other in X-direction; 
 use the first projection direction for areas of the two-dimensional projection plane which have a same Y-coordinate as a first area of the two-dimensional projection plane; and 
 use the second projection direction for areas of the two-dimensional projection plane which have a same Y-coordinate as a second area of the two-dimensional projection plane. 
 
     
     
       7. The apparatus according to  claim 1 , said memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
 encode the first and second projection directions in an auxiliary patch information. 
 
     
     
       8. The apparatus according to  claim 7 , wherein the auxiliary patch information comprises one or more of the following:
 index of the projection plane; 
 two-dimensional (2D) bounding box; or 
 three-dimensional (3D) location of the patch represented in terms of depth, tangential shift and bi-tangential shift. 
 
     
     
       9. The apparatus according to  claim 1 , wherein the first concentration of the at least one first projection direction is equal to the second concentration of the at least one second projection direction, in response to the first quality level being equal to the second quality level. 
     
     
       10. The apparatus according to  claim 1 , wherein the first concentration of the at least one first projection direction for the first part of the two-dimensional projection plane, and the second concentration of the at least one second projection direction for the second part of the two-dimensional projection plane are determined based on at least one of:
 a region of interest within the scene; 
 an amount of motion within the scene; 
 a distance of an object from the two-dimensional projection plane; 
 a complexity of the object; or 
 a spatial resolution of the two-dimensional projection plane. 
 
     
     
       11. The apparatus according to  claim 1 ,
 wherein the first concentration of the at least one first projection direction, and projection directions of the at least one first projection direction adaptively change across parts of the scene depending on at least of: a region of interest within a part of the scene; an amount of motion within the part of the scene; a distance of an object from the two-dimensional projection plane within the part of the scene; a complexity of the object within the part of the scene; or a spatial resolution of the two-dimensional projection plane; and 
 wherein the second concentration of the at least one second projection direction, and projection directions of the at least one second projection direction adaptively change across parts of the scene depending on at least one of: a region of interest within the part of the scene; an amount of motion within the part of the scene; a distance of an object from the two-dimensional projection plane within the part of the scene; a complexity of the object within the part of the scene; or the spatial resolution of the two-dimensional projection plane. 
 
     
     
       12. The apparatus according to  claim 1 , wherein the second concentration of the at least one second projection direction is higher than the first concentration of the at least one first projection direction, in response to the second quality level being higher than the first quality level. 
     
     
       13. An apparatus comprising at least one processor, non-transitory memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
 decode information of a two-dimensional projection plane of a three-dimensional scene; 
 decode information of a first concentration of at least one first projection direction for a first part of the two-dimensional projection plane, based on a determined first quality level to project a first part of the scene; 
 decode information of a second concentration of at least one second projection direction for a second part of the two-dimensional projection plane, based on a determined second quality level to project a second part of the scene; 
 wherein the first concentration of the at least one first projection direction is higher than the second concentration of the at least one second projection direction, in response to the first quality level being higher than the second quality level; 
 wherein the first projection direction for the first part of the two-dimensional projection plane is different from the second projection direction for the second part of the two-dimensional projection plane; 
 use the decoded information to obtain reconstructed geometry data of different parts of the two-dimensional projection plane; and 
 use the reconstructed geometry data to form a reconstructed three-dimensional image. 
 
     
     
       14. The apparatus according to  claim 13 ,
 wherein the information of the first projection direction and the second projection direction is encoded as a function of location on the two-dimensional projection plane and at least one residual between an output of the function and the first projection direction or the second projection direction; and 
 wherein the function encodes a distribution of projection directions of the at least one first projection direction and projection directions of the at least one second projection direction. 
 
     
     
       15. The apparatus according to  claim 13 , said memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
 decode said information of the first projection direction as a difference between the first projection direction and a reference direction. 
 
     
     
       16. The apparatus according to  claim 15 , wherein at least one of the first projection or the second projection direction are temporally predicted from at least one previous temporal instance of the two-dimensional projection plane. 
     
     
       17. The apparatus according to  claim 15 , said memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to decode said information of the second projection direction as one of the following:
 a difference between the second projection direction and the reference direction; or 
 a difference between the second projection direction and the first projection direction. 
 
     
     
       18. The apparatus according to  claim 13 , said memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
 use the same first projection direction for areas of the two-dimensional projection plane which have a same Y-coordinate as a first area of the two-dimensional projection plane; and 
 use the same second projection direction for areas of the two-dimensional projection plane which have a same Y-coordinate as a second area of the two-dimensional projection plane. 
 
     
     
       19. A method, comprising:
 obtaining information of a three-dimensional scene; 
 determining a two-dimensional projection plane for the scene; 
 determining a first concentration of at least one first projection direction for a first part of the two-dimensional projection plane, based on a determined first quality level to project a first part of the scene; 
 determine a second concentration of at least one second projection direction for a second part of the two-dimensional projection plane, based on determined second quality level to project a second part of the scene; 
 wherein the first concentration of the at least one first projection direction is higher than the second concentration of the at least one second projection direction, in response to the first quality level being higher than the second quality level; 
 wherein the first projection direction for the first part of the two-dimensional projection plane is different from the second projection direction for the second part of the two-dimensional projection plane; 
 projecting a first part of the scene on the two-dimensional projection plane from the first projection direction; 
 projecting a second part of the scene on the two-dimensional projection plane from the second projection direction; and 
 encoding the two-dimensional projection plane. 
 
     
     
       20. The method according to  claim 19 , further comprising:
 encoding information of the at least one first projection direction and the at least one second projection direction; and 
 signal the information of the at least one first projection direction and the at least one second projection direction to a decoder; 
 wherein the information encoded and signaled comprises the first concentration of the at least one first projection direction, and the second concentration of the at least one second projection direction. 
 
     
     
       21. A method, comprising:
 decoding information of a two-dimensional projection plane of a three-dimensional scene; 
 decoding information of a first concentration of at least one first projection direction for a first part of the two-dimensional projection plane, based on a determined first quality level to project a first part of the scene; 
 decoding information of a second concentration of at least one second projection direction for a second part of the two-dimensional projection plane, based on a determined second quality level to project a second part of the scene; 
 wherein the first concentration of the at least one first projection direction is higher than the second concentration of the at least one second projection direction, in response to the first quality level being higher than the second quality level; 
 wherein the first projection direction for the first part of the two-dimensional projection plane is different from the second projection direction for the second part of the two-dimensional projection plane; 
 using the decoded information to obtain reconstructed geometry data of different parts of the two-dimensional projection plane; and 
 using the reconstructed geometry data to form a reconstructed three-dimensional image. 
 
     
     
       22. The method according to  claim 21 , wherein the information of the first projection direction and the second projection direction is encoded as a function of location on the two-dimensional projection plane and at least one residual between an output of the function and the first projection direction or the second projection direction. 
     
     
       23. A non-transitory computer readable storage medium comprising code for use by an apparatus, which when executed by a processor, causes the apparatus to perform:
 obtain information of a three-dimensional scene; 
 determine at least one two-dimensional projection plane for the scene; 
 determine a first concentration of at least one first projection direction for a first part of the two-dimensional projection plane, based on a determined first quality level to project a first part of the scene; 
 determine a second concentration of at least one second projection direction for a second part of the two-dimensional projection plane, based on a determined second quality level to project a second part of the scene; 
 wherein the first projection direction for the first part of the two-dimensional projection plane is different from the second projection direction for the second part of the two-dimensional projection plane; 
 project a first part of the scene on the two-dimensional projection plane from the first projection direction; 
 project a second part of the scene on the two-dimensional projection plane from the second projection direction; and 
 encode the two-dimensional projection plane. 
 
     
     
       24. The non-transitory computer readable storage medium according to  claim 23 , wherein the first concentration of the at least one first projection direction is equal to the second concentration of the at least one second projection direction, in response to the first quality level being equal to or substantially equal to the second quality level. 
     
     
       25. A non-transitory computer readable storage medium comprising code for use by an apparatus, which when executed by a processor, causes the apparatus to perform:
 decode information of a two-dimensional projection plane of a three-dimensional scene; 
 decode information of a first concentration of at least one first projection direction for a first part of the two-dimensional projection plane, based on a determined first quality level to project a first part of the scene; 
 decode information of a second concentration of at least one second projection direction for a second part of the two-dimensional projection plane, based on a determined second quality level to project a second part of the scene; 
 wherein the first projection direction for the first part of the two-dimensional projection plane is different from the second projection direction for the second part of the two-dimensional projection plane; 
 use the decoded information to obtain reconstructed geometry data of different parts of the two-dimensional projection plane; and 
 use the reconstructed geometry data to form a reconstructed three-dimensional image.