Keypoints based video compression

Abstract

Methods and apparatuses are provided for compressing video data based on keypoint features. An exemplary video compression method includes: receiving a video sequence; encoding one or more pictures of the video sequence; and generating a bitstream; wherein the encoding includes: representing a first picture by a first set of keypoints and a second set of keypoints, the second set comprising less keypoints than the first set; and compressing the video sequence based on the first set and second set of keypoints.

What is claimed is: 
     
         1 . A video compression method, comprising:
 receiving a video sequence;   encoding one or more pictures of the video sequence, wherein the encoding comprises:
 representing a first picture by a first set of keypoints and a second set of keypoints, the second set comprising less keypoints than the first set, the first set of keypoints comprising canonical keypoints extracted from a second picture; and 
 compressing the video sequence based on the first set and second set of keypoints; and 
   generating a bitstream associated with the compressed video sequence.   
     
     
         2 . The video compression method of  claim 1 , wherein the second set of keypoints is a subset of the first set of keypoints, and representing the first picture by the first set of keypoints and the second set of keypoints comprises:
 transforming the first set of keypoints;   determining a deformation to the second set of keypoints; and   representing the first picture based on the transformed first set of keypoints and the deformation to the second set of keypoints.   
     
     
         3 . The video compression method of  claim 2 , wherein transforming the first set of keypoints comprises performing at least one of a rotation or a translation of the first set of keypoints. 
     
     
         4 . The video compression method of  claim 1 , wherein the first and second pictures comprises a representation of a face, and the canonical keypoints are independent of a pose and an expression of the face. 
     
     
         5 . The video compression method of  claim 1 , wherein the first picture comprises a representation of a face, and representing the first picture by the first set of keypoints and the second set of keypoints comprises:
 representing a pose of the face based on the first set of keypoints; and   representing an expression of the face based on the second set of keypoints.   
     
     
         6 . The video compression method of  claim 1 , wherein the encoding further comprises:
 generating, based on the first and second sets of keypoints, a reconstructed picture associated with the first picture; and   determining a quality loss of the reconstructed picture, based on a weighted combination of at least two models, wherein the at least two models are trained on different training datasets.   
     
     
         7 . The video compression method of  claim 6 , wherein at least one of the training datasets comprises face pictures. 
     
     
         8 . A video decoding method, comprising:
 receiving a bitstream; and   decoding, using coded information of the bitstream, one or more pictures, wherein the decoding comprises:
 decoding warping data associated with a first set of keypoints and a second set of keypoints, the second set comprising less keypoints than the first set, the first set of keypoints comprising canonical keypoints extracted from a second picture; and 
 generating a first picture based on the first set of keypoints, the second set of keypoints, and the warping data. 
   
     
     
         9 . The video decoding method of  claim 8 , wherein the second set of keypoints is a subset of the first set of keypoints. 
     
     
         10 . The video decoding method of  claim 8 , wherein generating the first picture based on the first set of keypoints, the second set of keypoints, and the warping data comprises:
 warping one or more features of a second picture, based on the warping data; and   generating the first picture based on the warped one or more features.   
     
     
         11 . The video decoding method of  claim 10 , wherein the first and second pictures comprises a representation of a face, and the canonical keypoints are independent of a pose and an expression of the face. 
     
     
         12 . The video decoding method of  claim 8 , wherein the first picture comprises a representation of a face, and generating the first picture based on the first set of keypoints, the second set of keypoints, and the warping data comprises:
 generating a pose of the face based on the first set of keypoints; and   generating an expression of the face based on the second set of keypoints.   
     
     
         13 . The video decoding method of  claim 8 , further comprising:
 generating, based on the first and second sets of keypoints, a reconstructed picture associated with the first picture; and   determining a quality loss of the reconstructed picture, based on a weighted combination of at least two models, wherein the at least two models are trained on different training datasets.   
     
     
         14 . The video decoding method of  claim 13 , wherein at least one of the training datasets comprises face pictures. 
     
     
         15 . A method of storing a bitstream of a video, the method comprising:
 generating a first set of keypoints and a second set of keypoints representing a first picture, the second set comprising less keypoints than the first set, the first set of keypoints comprising canonical keypoints extracted from a second picture;   generating warping data associated with the first and second sets of keypoints, wherein the first set of keypoints, the second set of keypoints, and the warping data are used for generating the first picture;   generating a bitstream comprising the warping data; and   storing the bitstream in a non-transitory computer readable storage medium.   
     
     
         16 . The method of  claim 15 , wherein the second set of keypoints is a subset of the first set of keypoints, and the first picture is represented by a combination of:
 a transformation of the first set of keypoints; and   a deformation to the second set of keypoints.   
     
     
         17 . The method of  claim 16 , wherein the transformation of the first set of keypoints comprises at least one of a rotation or a translation of the first set of keypoints.