Dynamic local temporal-consistent textured mesh compression

Abstract

Mesh-based raw video data (or 3D video data) includes a sequence of frames, each of which includes geometry data (e.g., triangle meshes or other meshes) and texture map(s) defining one or more objects. The raw 3D video data is segmented based on consistent mesh topology across frames. For each segment, a consistent mesh sequence (CMS) is defined and a consistent texture atlas (CTA) is generated. The CMS and CTA for each segment are compressed and stored as compressed data files. The compressed data files can be decompressed and used to render displayable images.

What is claimed is: 
     
       1. A computer-implemented method for compressing data, the method comprising:
 receiving raw three-dimensional (3D) video data descriptive of a sequence of frames, wherein the raw 3D video data for each frame includes a mesh and a texture map associated with the mesh; 
 splitting the sequence of frames into one or more segments such that the meshes for all frames within a segment are topologically consistent, thereby generating a consistent mesh sequence for each segment; 
 constructing a consistent texture atlas for each segment from the texture map for at least one of the frames in the segment; 
 compressing the consistent texture atlas using a two-dimensional (2D) video compression algorithm; 
 compressing the consistent mesh sequence; and 
 outputting, as compressed data, the compressed texture atlas and the compressed consistent mesh sequence, wherein the raw 3D video data does not include data indicating correlations between meshes in different frames. 
 
     
     
       2. The method of  claim 1  wherein splitting the sequence of frames includes:
 selecting one of the frames as a keyframe; and 
 determining segment boundaries based on a deformation error determined by deforming the mesh from the keyframe to conform to the mesh of another frame. 
 
     
     
       3. The method of  claim 2  wherein deforming the mesh from the keyframe to conform to the mesh of another frame includes applying a non-rigid registration algorithm to identify corresponding vertices in the keyframe and the other frame. 
     
     
       4. The method of  claim 1  wherein constructing the consistent texture atlas includes reprojecting the texture coordinates of each vertex of the mesh. 
     
     
       5. The method of  claim 1  wherein compressing the consistent mesh sequence for each segment includes:
 determining a trajectory for each vertex within the segment; 
 defining clusters of vertices based the trajectories of the vertices; 
 constructing a matrix of cluster trajectories; 
 performing principal component analysis on the matrix of cluster trajectories; and 
 storing a set of eigenvectors and coefficients resulting from the principal component analysis. 
 
     
     
       6. The method of  claim 1  wherein outputting the compressed texture atlas and the compressed consistent mesh sequence includes storing a data file that includes the compressed texture atlas and the compressed consistent mesh sequence. 
     
     
       7. The method of  claim 1  wherein outputting the compressed texture atlas and the compressed consistent mesh sequence includes streaming the compressed texture atlas and the compressed consistent mesh sequence to a receiving computer system. 
     
     
       8. The method of  claim 7  wherein outputting the compressed texture atlas and the compressed consistent mesh sequence further includes streaming an audio file to the receiving computer system, wherein the audio file is time-aligned with the compressed texture atlas and the compressed consistent mesh sequence. 
     
     
       9. A system comprising:
 a processing subsystem; and 
 a storage subsystem coupled to the processing subsystem, the storage subsystem storing program code instructions that, when executed by the processing subsystem, cause the processing subsystem to: 
 receive raw three-dimensional (3D) video data descriptive of a sequence of frames, wherein the raw 3D video data for each frame includes a mesh and a texture map associated with the mesh; 
 split the sequence of frames into one or more segments such that the meshes for all frames within a segment are topologically consistent, thereby generating a consistent mesh sequence for each segment; 
 construct a consistent texture atlas for each segment from the texture map for at least one of the frames in the segment; 
 compress the consistent texture atlas using a two-dimensional (2D) video compression algorithm; 
 compress the consistent mesh sequence; and 
 output, as compressed data, the compressed texture atlas and the compressed consistent mesh sequence, wherein the program code instructions include instructions such that compressing the consistent mesh sequence for each segment includes:
 determining a trajectory for each vertex within the segment; 
 defining clusters of vertices based the trajectories of the vertices; 
 constructing a matrix of cluster trajectories; 
 performing principal component analysis on the matrix of cluster trajectories; and 
 storing a set of eigenvectors and coefficients resulting from the principal component analysis. 
 
 
     
     
       10. The system of  claim 9  wherein the raw 3D video data does not include data indicating correlations between meshes in different frames. 
     
     
       11. The system of  claim 9  wherein the program code instructions include instructions such that splitting the sequence of frames includes:
 selecting one of the frames as a keyframe; and 
 determining segment boundaries based on a deformation error determined by deforming the mesh from the keyframe to conform to the mesh of another frame. 
 
     
     
       12. The system of  claim 11  wherein deforming the mesh from the keyframe to conform to the mesh of another frame includes applying a non-rigid registration algorithm to identify corresponding vertices in the keyframe and the other frame. 
     
     
       13. The system of  claim 9  wherein the program code instructions include instructions such that constructing the consistent texture atlas includes reprojecting the texture coordinates of each vertex of the mesh. 
     
     
       14. The system of  claim 9  wherein the program code instructions include instructions such that outputting the compressed texture atlas and the compressed consistent mesh sequence includes storing a data file that includes the compressed texture atlas and the compressed consistent mesh sequence. 
     
     
       15. The system of  claim 9  wherein the program code instructions include instructions such that outputting the compressed texture atlas and the compressed consistent mesh sequence includes streaming the compressed texture atlas and the compressed consistent mesh sequence to a receiving computer system. 
     
     
       16. A non-transitory computer-readable storage medium having stored thereon program code that, when executed by a processor of a computer system, cause the processor to perform a method comprising:
 receiving raw three-dimensional (3D) video data descriptive of a sequence of frames, wherein the raw 3D video data for each frame includes a mesh and a texture map associated with the mesh; 
 splitting the sequence of frames into one or more segments such that the meshes for all frames within a segment are topologically consistent, thereby generating a consistent mesh sequence for each segment; 
 constructing a consistent texture atlas for each segment from the texture map for at least one of the frames in the segment; 
 compressing the consistent texture atlas using a two-dimensional (2D) video compression algorithm; 
 compressing the consistent mesh sequence; and 
 outputting, as compressed data, the compressed texture atlas and the compressed consistent mesh sequence, wherein outputting the compressed texture atlas and the compressed consistent mesh sequence includes storing a data file that includes the compressed texture atlas and the compressed consistent mesh sequence. 
 
     
     
       17. The non-transitory computer-readable storage medium of  claim 16  wherein the raw 3D video data does not include data indicating correlations between meshes in different frames. 
     
     
       18. The non-transitory computer-readable storage medium of  claim 16  wherein splitting the sequence of frames includes:
 selecting one of the frames as a keyframe; and 
 determining segment boundaries based on a deformation error determined by deforming the mesh from the keyframe to conform to the mesh of another frame. 
 
     
     
       19. The non-transitory computer-readable storage medium of  claim 18  wherein deforming the mesh from the keyframe to conform to the mesh of another frame includes applying a non-rigid registration algorithm to identify corresponding vertices in the keyframe and the other frame. 
     
     
       20. The non-transitory computer-readable storage medium of  claim 16  wherein constructing the consistent texture atlas includes reprojecting the texture coordinates of each vertex of the mesh. 
     
     
       21. The non-transitory computer-readable storage medium of  claim 16  wherein compressing the consistent mesh sequence for each segment includes:
 determining a trajectory for each vertex within the segment; 
 defining clusters of vertices based the trajectories of the vertices; 
 constructing a matrix of cluster trajectories; 
 performing principal component analysis on the matrix of cluster trajectories; and 
 storing a set of eigenvectors and coefficients resulting from the principal component analysis. 
 
     
     
       22. The non-transitory computer-readable storage medium of  claim 16  wherein outputting the compressed texture atlas and the compressed consistent mesh sequence includes storing a data file that includes the compressed texture atlas and the compressed consistent mesh sequence. 
     
     
       23. The non-transitory computer-readable storage medium of  claim 16  wherein outputting the compressed texture atlas and the compressed consistent mesh sequence includes streaming the compressed texture atlas and the compressed consistent mesh sequence to a receiving computer system. 
     
     
       24. The non-transitory computer-readable storage medium of  claim 23  wherein outputting the compressed texture atlas and the compressed consistent mesh sequence further includes streaming an audio file to the receiving computer system, wherein the audio file is time-aligned with the compressed texture atlas and the compressed consistent mesh sequence. 
     
     
       25. A computer-implemented method for compressing data, the method comprising:
 receiving raw three-dimensional (3D) video data descriptive of a sequence of frames, wherein the raw 3D video data for each frame includes a mesh and a texture map associated with the mesh; 
 splitting the sequence of frames into one or more segments such that the meshes for all frames within a segment are topologically consistent, thereby generating a consistent mesh sequence for each segment; 
 constructing a consistent texture atlas for each segment from the texture map for at least one of the frames in the segment; 
 compressing the consistent texture atlas using a two-dimensional (2D) video compression algorithm; 
 compressing the consistent mesh sequence; and 
 outputting, as compressed data, the compressed texture atlas and the compressed consistent mesh sequence, wherein outputting the compressed texture atlas and the compressed consistent mesh sequence includes streaming the compressed texture atlas and the compressed consistent mesh sequence to a receiving computer system.