Apparatus, a method and a computer program for volumetric video

Abstract

Embodiments for volumetric video encoding and decoding relating to one or more three-dimensional objects are disclosed. In encoding, after mapping from 3D space to 2D plane (802) a point in the 2D plane is examined (805) to determine which points of the 3D object are mapped to the same point to obtain a set of candidate points. Candidate points belonging to a same surface can be used to determine a center of mass for the surface (807). A depth value of the centre of mass is mapped to a 2D projection depth plane (808). A colour value for the centre of mass is interpolated from colour values of points of the set of surface points which are nearest neighbours of the center of mass (810), and used as the colour of the surface in the texture plane (812). Corresponding embodiments for decoding are provided.

The invention claimed is: 
     
       1. A method comprising:
 receiving coded texture data representing a texture image of a three-dimensional object in volumetric video data, wherein the coded texture data comprises an interpolated colour value for a same surface of the three-dimensional object, wherein the interpolated colour value comprises a color value interpolated for a determined centre of mass for the same surface; 
 receiving coded depth data representing a depth image of the three-dimensional object in the volumetric video data; 
 selecting a first pixel from one of the texture image or the depth image; 
 selecting a second pixel from one of the texture image or the depth image; 
 comparing coded depth data of the first pixel with coded depth data of the second pixel to determine whether the first pixel and the second pixel belong to the same surface of the three-dimensional object, wherein the coded depth data of the first pixel comprises a first depth value from the depth image, wherein the coded depth data of the second pixel comprises a second depth value from the depth image; 
 determining the first pixel and the second pixel belong to the same surface of the three-dimensional object in response to the coded depth data of the first pixel and the coded depth data of the second pixel being the same or almost the same; 
 upsampling coordinates of the first pixel to obtain a first point in a three-dimensional space; 
 upsampling coordinates of the second pixel to obtain a second point in the three-dimensional space; and 
 reconstructing pixels between the first point and the second point in the three-dimensional space, comprising:
 interpolating texture values of the first pixel and the second pixel in the texture image based, at least partially, on the interpolated colour value for the same surface of the three-dimensional object; and 
 interpolating depth values of the first pixel and the second pixel in the depth image. 
 
 
     
     
       2. The method according to  claim 1  further comprising:
 selecting the second pixel among one or more pixels which are neighbours of said first pixel. 
 
     
     
       3. The method according to  claim 1  further comprising:
 using a different interpolation factor in the depth image and in the texture image. 
 
     
     
       4. The method according to  claim 1 , wherein receiving the coded texture data representing the texture image comprises:
 obtaining the volumetric video data containing information of one or more three-dimensional objects, wherein the one or more three-dimensional objects comprises, at least, the three-dimensional object; 
 mapping the one or more three-dimensional objects to a two-dimensional projection plane with a pre-determined vertical and horizontal resolution; 
 examining a point in the two-dimensional projection plane to determine which points of the three-dimensional object are mapped to the point to obtain a set of candidate points; 
 determining which points of the set of candidate points belong to the same surface to obtain a set of surface points; 
 determining the centre of mass for the same surface based, at least partially, on points of the set of surface points; 
 mapping a depth value of the centre of mass to the depth image, wherein the depth image comprises a representation of a two-dimensional projection plane of depth data of the three-dimensional object; 
 interpolating the colour value for the determined centre of mass from colour values of the points of the set of surface points which are nearest neighbours of the centre of mass; and 
 using the interpolated colour value as the colour of the same surface in the texture image. 
 
     
     
       5. The method according to  claim 4  further comprising:
 obtaining the set of candidate points via back-projection of the two-dimensional projection plane with the pre-determined vertical and horizontal resolution to a three-dimensional plane; and 
 using a point obtained via the back-projection to construct the set of candidate points. 
 
     
     
       6. The method according to  claim 5 , wherein the point obtained via the back-projection is an outmost point projected onto two-dimensional coordinates. 
     
     
       7. The method according to  claim 4  further comprising:
 dividing the same surface into four quadratic regions. 
 
     
     
       8. An apparatus comprising:
 at least one processor and 
 at least one non-transitory memory, 
 said at least one memory stored with code thereon, which when executed by said at least one processor, causes the apparatus to perform at least:
 receive coded texture data representing a texture image of a three-dimensional object in volumetric video data, wherein the coded texture data comprises an interpolated colour value for a same surface of the three-dimensional object, wherein the interpolated colour value comprises 
 a color value interpolated for a determined centre of mass for the same surface; 
 receive coded depth data representing a depth image of the three-dimensional object in the volumetric video data; 
 select a first pixel from one of the texture image or the depth image; 
 select a second pixel from one of the texture image or the depth image; 
 compare coded depth data of the first pixel with coded depth data of the second pixel to determine whether the first pixel and the second pixel belong to the same surface of the three-dimensional object, wherein the coded depth data of the first pixel comprises a first depth value from the depth image, wherein the coded depth data of the second pixel comprises a second depth value from the depth image; 
 determine the first pixel and the second pixel belong to the same surface of the three-dimensional object in response to the coded depth data of the first pixel and the coded depth data of the second pixel being the same or almost the same; 
 upsample coordinates of the first pixel to obtain a first point in a three-dimensional space; 
 upsample coordinates of the second pixel to obtain a second point in the three-dimensional space; and 
 reconstruct pixels between the first point and the second point in the three-dimensional space, comprising:
 interpolating texture values of the first pixel and the second pixel in the texture image based, at least partially, on the interpolated colour value for the same surface of the three-dimensional object; and 
 interpolating depth values of the first pixel and the second pixel in the depth image. 
 
 
 
     
     
       9. The apparatus according to  claim 8 , wherein the apparatus is further caused to:
 select the second pixel among one or more pixels which are neighbours of said first pixel. 
 
     
     
       10. The apparatus according to  claim 8 , wherein the apparatus is further caused to:
 use a different interpolation factor in the depth image and in the texture image. 
 
     
     
       11. The apparatus according to  claim 8 , wherein receiving the coded texture data representing the texture image further causes the apparatus to:
 obtain the volumetric video data containing information of one or more three-dimensional objects, wherein the one or more three-dimensional objects comprises, at least, the three-dimensional object; 
 map the one or more three-dimensional objects to a two-dimensional projection plane with a pre-determined vertical and horizontal resolution; 
 examine a point in the two-dimensional projection plane to determine which points of the three-dimensional object are mapped to the point to obtain a set of candidate points; 
 determine which points of the set of candidate points belong to the same surface to obtain a set of surface points; 
 determine the centre of mass for the same surface based, at least partially, on points of the set of surface points; 
 map a depth value of the centre of mass to the depth image, wherein the depth image is configured to represent a two-dimensional projection plane of depth data of the three-dimensional object; 
 interpolate the colour value for the determined centre of mass from colour values of points of the set of surface points which are nearest neighbours of the centre of mass; and 
 use the interpolated colour value as the colour of the same surface in the texture image. 
 
     
     
       12. The apparatus according to  claim 11 , wherein the apparatus is further caused to:
 obtain the set of candidate points via back-projection of the two-dimensional projection plane with the pre-determined vertical and horizontal resolution to a three-dimensional plane; and 
 use a point obtained via the back-projection to construct the set of candidate points. 
 
     
     
       13. The apparatus according to  claim 11 , wherein the point obtained via the back-projection is an outmost point projected onto two-dimensional coordinates. 
     
     
       14. The apparatus according to  claim 11 , wherein the apparatus is further caused to:
 divide the same surface into four quadratic regions. 
 
     
     
       15. The apparatus according to  claim 14 , wherein the apparatus is further caused to:
 determine in which of the four quadratic regions points in the set of surface points belongs, and 
 determine which point of the points is closest to the centre of mass for respective quadratic regions of the determined quadratic regions, 
 wherein the interpolation of the colour value comprises using closest points of the determined regions. 
 
     
     
       16. The apparatus according to  claim 15 , wherein the apparatus is further caused to:
 determine the colour value of the centre of mass using distances of the closest points to the centre of mass of the same surface in the interpolation. 
 
     
     
       17. The apparatus according to  claim 11 , wherein the apparatus is further caused to:
 select a third pixel from the set of candidate points; 
 determine a three-dimensional location of the third pixel in the volumetric video data; 
 select a fourth pixel from the set of candidate points; 
 determine a three-dimensional location of the fourth pixel in the volumetric video data; 
 determine a gradient on the basis of the three-dimensional location of the third pixel and the three-dimensional location of the fourth pixel; 
 compare the gradient with a threshold; and 
 exclude the fourth pixel from the set of surface points responsive to the gradient being greater than the threshold. 
 
     
     
       18. The apparatus according to  claim 11 , wherein a resolution of the depth image is lower than a resolution of the texture image. 
     
     
       19. A non-transitory computer-readable medium comprising program instructions stored thereon which, when executed with at least one processor, cause the at least one processor to:
 receive coded texture data representing a texture image of a three-dimensional object in volumetric video data, wherein the coded texture data comprises an interpolated colour value for a same surface of the three-dimensional object, wherein the interpolated colour value comprises a color value interpolated for a determined centre of mass for the same surface; 
 receive coded depth data representing a depth image of the three-dimensional object in the volumetric video data; 
 cause selection of a first pixel from one of the texture image or the depth image; 
 cause selection of a second pixel from one of the texture image or the depth image; 
 compare coded depth data of the first pixel with coded depth data of the second pixel to determine whether the first pixel and the second pixel belong to the same surface of the three-dimensional object, wherein the coded depth data of the first pixel comprises a first depth value from the depth image, wherein the coded depth data of the second pixel comprises a second depth value from the depth image; 
 determine the first pixel and the second pixel belong to the same surface of the three-dimensional object in response to the coded depth data of the first pixel and the coded depth data of the second pixel being the same or almost the same; 
 cause upsampling of coordinates of the first pixel to obtain a first point in a three-dimensional space; 
 cause upsampling of coordinates of the second pixel to obtain a second point in the three-dimensional space; and 
 cause reconstruction of pixels between the first point and the second point in the three-dimensional space, comprising:
 cause interpolating of texture values of the first pixel and the second pixel in the texture image based, at least partially, on the interpolated colour value for the same surface of the three-dimensional object; and 
 cause interpolating of depth values of the first pixel and the second pixel in the depth image.