Techniques for transform kernel set selection/derivation

Abstract

An aspect of the disclosure provides a method of video decoding. For example, a coded video bitstream is received. The coded video bitstream includes coded information of a plurality of pictures. Based on coded information of a current block in a current picture, it is determined that the current block is coded using an intra prediction that generates prediction samples of the current block based on reference samples in the current picture. At least a first transform kernel set is determined based on a histogram of occurrence (HoC) of intra prediction modes in neighboring samples of the current block. Based on the coded information of the current block, a residual block of the current block is calculated according to at least the first transform kernel set. The current block is reconstructed based on the residual block and the intra prediction of the current block.

What is claimed is: 
     
         1 . A method of video decoding, comprising:
 receiving a coded video bitstream comprising coded information of a plurality of pictures;   determining, based on coded information of a current block in a current picture, that the current block is coded using an intra prediction that generates prediction samples of the current block based on reference samples in the current picture;   determining at least a first transform kernel set based on a histogram of occurrence (HoC) of intra prediction modes in neighboring samples of the current block;   calculating, based on the coded information of the current block, a residual block of the current block according to at least the first transform kernel set; and   reconstructing the current block based on the residual block and the intra prediction of the current block.   
     
     
         2 . The method of  claim 1 , wherein the determining at least the first transform kernel set comprises:
 determining at least a first intra prediction mode based on the HoC of the intra prediction modes in the neighboring samples of the current block; and   determining the first transform kernel set that maps to the first intra prediction mode according to a one-to-one mapping of the intra prediction modes to transform kernel sets.   
     
     
         3 . The method of  claim 2 , wherein the determining at least the first intra prediction mode comprises:
 determining the first intra prediction mode that has a highest occurrence in the intra prediction modes of the HoC.   
     
     
         4 . The method of  claim 2 , wherein the determining at least the first intra prediction mode comprises:
 determining the first intra prediction mode that has a second highest occurrence in the intra prediction modes of the HoC.   
     
     
         5 . The method of  claim 2 , wherein the determining at least the first intra prediction mode comprises:
 determining the first intra prediction mode that has an occurrence value higher than an occurrence threshold according to the HoC.   
     
     
         6 . The method of  claim 2 , wherein the first intra prediction mode is a planar mode that is predefined. 
     
     
         7 . The method of  claim 2 , wherein the determining at least the first intra prediction mode comprises:
 determining a highest occurrence mode that has a highest occurrence in the intra prediction modes of the HoC; and   determining the first intra prediction mode by applying an offset to the highest occurrence mode.   
     
     
         8 . The method of  claim 1 , wherein the determining at least the first transform kernel set comprises:
 determining at least a first intra prediction mode and a second intra prediction mode based on the HoC of the intra prediction modes in the neighboring samples of the current block; and   determining the first transform kernel set and a second transform kernel that respectively map to the first intra prediction mode and the second intra prediction mode according to a one-to-one mapping of the intra prediction modes to transform kernel sets.   
     
     
         9 . The method of  claim 1 , wherein the determining at least the first transform kernel set comprises:
 constructing a most probable transform kernel candidate list that includes a plurality of transform kernel sets based on the HoC of the intra prediction modes in the neighboring samples of the current block.   
     
     
         10 . The method of  claim 1 , further comprising:
 decoding a syntax element from the coded information of the current block, the syntax element indicating a selected transform kernel set from at least the first transform kernel set, the selected transform kernel set being used for the calculating of the residual block of the current block.   
     
     
         11 . A method of video decoding, comprising:
 receiving a coded video bitstream comprising coded information of a plurality of pictures;   determining, based on coded information of a current block in a current picture, that the current block is coded in an adaptive transform kernel derivation mode with intra and inter partitions, the current block being partitioned into at least a first partition and a second partition, the first partition being predicted based on a vector that points to a reference block and the second partition being predicted by an intra prediction;   determining an intra prediction mode from a plurality of candidate intra prediction modes for the current block based on the coded information of the current block;   determining a transform kernel set based on the intra prediction mode;   calculating a residual block of the current block according to the transform kernel set and the coded information of the current block; and   reconstructing the current block based on the residual block, the reference block of the first partition and the intra prediction of the second partition.   
     
     
         12 . The method of  claim 11 , wherein the current block is partitioned with a non-rectangular boundary. 
     
     
         13 . The method of  claim 11 , wherein the current block is partitioned with a rectangular boundary. 
     
     
         14 . The method of  claim 11 , wherein the plurality of candidate intra prediction modes include a first candidate intra prediction mode associated with a partition direction of the current block. 
     
     
         15 . The method of  claim 14 , further comprising:
 determining the first candidate intra prediction mode according to a predefined lookup table that maps partition directions to intra prediction modes.   
     
     
         16 . The method of  claim 11 , wherein the plurality of candidate intra prediction modes include a second candidate intra prediction mode that is derived based on a histogram of gradients associated with the current block. 
     
     
         17 . The method of  claim 16 , further comprising:
 building the histogram of gradients according to predicted samples of the current block, the gradients being mapped to intra prediction modes;   ranking the intra prediction modes based on the histogram of gradients; and   determining the second candidate intra prediction mode according to the intra prediction modes that are ranked based on the histogram of gradients.   
     
     
         18 . The method of  claim 17 , wherein the second candidate intra prediction mode is a most frequent mode based on the histogram of gradients. 
     
     
         19 . The method of  claim 17 , wherein the second candidate intra prediction mode is a second most frequent mode based on the histogram of gradients. 
     
     
         20 . The method of  claim 11 , wherein the plurality of candidate intra prediction modes include a first candidate intra prediction mode associated with a partition direction of the current block and a second candidate intra prediction mode that is derived based on a histogram of gradients associated with the current block, and the method comprises:
 selecting the intra prediction mode from the first candidate intra prediction mode and the second candidate intra prediction mode based on a similarity between the first candidate intra prediction mode and the second candidate intra prediction mode.