Video coding with fine granularity spatial scalability

Abstract

The disclosure is directed to video coding techniques that support spatial scalability using a generalized fine granularity scalability (FGS) approach. Various degrees of spatial scalability can be achieved by sending spatially scalable enhancement layers in a generalized FGS format. Spatially scalable enhancement bitstreams can be arbitrarily truncated to conform to network conditions, channel conditions and/or decoder capabilities. Coding coefficients and syntax elements for spatial scalability can be embedded in a generalized FGS format. For good network or channel conditions, and/or enhanced decoder capabilities, additional bits received via one or more enhancement layers permit encoded video to be reconstructed with increased spatial resolution and continuously improved video quality across different spatial resolutions. The techniques permit spatial scalability layers to be coded as FGS layers, rather than discrete layers, permitting arbitrary scalability. The techniques may include features to curb error propagation that may otherwise arise due to partial decoding.

1 . A method comprising:
 decoding a fine granularity scalability (FGS) base layer including base layer video blocks defining video information at a first spatial resolution;   at least partially decoding one or more FGS enhancement layers including enhancement layer video blocks defining video information at a second spatial resolution greater than the first spatial resolution;   determining a constraint that applies to a frame to be predicted from at least one of the decoded FGS base layer video blocks and the at least partially decoded one or more FGS enhancement layers, wherein the constraint indicates that intra-coded blocks of the frame are to be predicted without using video information derived from the same frame in the same layer as the respective intra-coded video blocks to be predicted; and   based on the determination of the constraint, predicting the intra-coded video blocks of the frame using at least one of the video information defined by the decoded FGS base layer and the video information defined by the at least partially decoded one or more FGS enhancement layers, excluding video information derived from the same frame in the same layer as the respective intra-coded video blocks of the frame.   
     
     
         2 . The method of  claim 1 , further comprising predicting the intra-coded video block using only pixels from the base layer. 
     
     
         3 . The method of  claim 1 , further comprising decoding a block in the FGS enhancement layer using a special DC mode in which a DC coefficient is predicted from a default value known to both an encoder and a decoder and AC coefficients are decoded without a prediction calculated from neighboring blocks. 
     
     
         4 . The method of  claim 1 , further comprising applying only a spatial direct mode, without using information from a reference frame, to an enhancement layer if the enhancement layer is coded in a bi-predictive (B) slice. 
     
     
         5 . The method of  claim 1 , further comprising upsampling a significance map from the base layer, and using the upsampled significance map to decode one or more of the enhancement layers. 
     
     
         6 . The method of  claim 1 , further comprising resetting a significance map from the base layer, and decoding coefficients only as zero coefficients or nonzero significant coefficients with no refinement coefficients in one or more of the enhancement layers. 
     
     
         7 . A device comprising a decoder that decodes a fine granularity scalability (FGS) base layer including base layer video blocks defining video information at a first spatial resolution, at least partially decodes one or more FGS enhancement layers including enhancement layer video blocks defining video information at a second spatial resolution greater than the first spatial resolution, determines a constraint that applies to a frame to be predicted from the decoded FGS base layer video blocks and the at least partially decoded one or more FGS enhancement layers, wherein the constraint indicates that intra-coded blocks of the frame are to be predicted without using video information derived from the same frame in the same layer as the respective intra-coded video blocks to be predicted, and based on the determination of the constraint, predicts intra-coded video blocks of the frame using at least one of the video information defined by the decoded FGS base layer and the video information defined by the at least partially decoded one or more FGS enhancement layers, excluding video information derived from the same frame in the same layer as the respective intra-coded video blocks of the frame. 
     
     
         8 . The device of  claim 7 , wherein the decoder predicts the intra-coded video block using only pixels from the base layer. 
     
     
         9 . The device of  claim 7 , wherein the decoder decodes a block in the FGS enhancement layer using a special DC mode in which a DC coefficient is predicted from a default value known to both an encoder and a decoder and decodes AC coefficients without a prediction calculated from neighboring blocks. 
     
     
         10 . The device of  claim 7 , wherein the decoder applies only a spatial direct mode, without using information from a reference frame, to an enhancement layer if the enhancement layer is coded in a bi-predictive (B) slice. 
     
     
         11 . The device of  claim 7 , wherein the decoder upsamples a significance map from the base layer, and uses the upsampled significance map to decode one or more of the enhancement layers. 
     
     
         12 . The device of  claim 7 , wherein the decoder resets a significance map from the base layer, and decodes coefficients only as zero coefficients or nonzero significant coefficients with no refinement coefficients in one or more of the enhancement layers. 
     
     
         13 . A device comprising:
 means for decoding a fine granularity scalability (FGS) base layer including base layer video blocks defining video information at a first spatial resolution;   means for at least partially decoding one or more FGS enhancement layers including enhancement layer video blocks defining video information at a second spatial resolution greater than the first spatial resolution;   means for determining a constraint that applies to a frame to be predicted from at least one of the decoded FGS base layer video blocks and the at least partially decoded one or more FGS enhancement layers, wherein the constraint indicates that intra-coded blocks of the frame are to be predicted without using video information derived from the same frame in the same layer as the respective intra-coded video blocks to be predicted; and   means for predicting, based on the determination of the constraint, the intra-coded video blocks of the frame using at least one of the video information defined by the decoded FGS base layer and the video information defined by the at least partially decoded one or more FGS enhancement layers, excluding video information derived from the same frame in the same layer as the respective intra-coded video blocks of the frame.   
     
     
         14 . The device of  claim 13 , further comprising means for predicting the intra-coded video block using only pixels from the base layer. 
     
     
         15 . The device of  claim 13 , further comprising means for decoding a block in the FGS enhancement layer using a special DC mode in which a DC coefficient is predicted from a default value known to both an encoder and a decoder and AC coefficients are decoded without a prediction calculated from neighboring blocks. 
     
     
         16 . The device of  claim 13 , further comprising means for applying only a spatial direct mode, without using information from a reference frame, to an enhancement layer if the enhancement layer is coded in a bi-predictive (B) slice. 
     
     
         17 . The device of  claim 13 , further comprising:
 means for upsampling a significance map from the base layer; and   means for using the upsampled significance map to decode one or more of the enhancement layers.   
     
     
         18 . The device of  claim 13 , further comprising:
 means for resetting a significance map from the base layer; and   means for decoding coefficients only as zero coefficients or nonzero significant coefficients with no refinement coefficients in one or more of the enhancement layers.   
     
     
         19 . A non-transitory computer-readable storage medium comprising instructions that, when executed, cause a processor to:
 decode a fine granularity scalability (FGS) base layer including base layer video blocks defining video information at a first spatial resolution;   at least partially decode one or more FGS enhancement layers including enhancement layer video blocks defining video information at a second spatial resolution greater than the first spatial resolution;   determine a constraint that applies to a frame to be predicted from at least one of the decoded FGS base layer video blocks and the at least partially decoded one or more FGS enhancement layers, wherein the constraint indicates that intra-coded blocks of the frame are to be predicted without using video information derived from the same frame in the same layer as the respective intra-coded video blocks to be predicted; and   based on the determination of the constraint, predict the intra-coded video blocks of the frame using at least one of the video information defined by the decoded FGS base layer and the video information defined by the at least partially decoded one or more FGS enhancement layers, excluding video information derived from the same frame in the same layer as the respective intra-coded video blocks of the frame.   
     
     
         20 . The non-transitory computer-readable storage medium of  claim 19 , further comprising instructions that cause the processor to predict the intra-coded video block using only pixels from the base layer. 
     
     
         21 . The non-transitory computer-readable storage medium of  claim 19 , further comprising instructions that cause the processor to decode a block in the FGS enhancement layer using a special DC mode in which a DC coefficient is predicted from a default value known to both an encoder and a decoder and AC coefficients are decoded without a prediction calculated from neighboring blocks. 
     
     
         22 . The non-transitory computer-readable storage medium of  claim 19 , further comprising instructions that cause the processor to apply only a spatial direct mode, without using information from a reference frame, to an enhancement layer if the enhancement layer is coded in a bi-predictive (B) slice. 
     
     
         23 . The non-transitory computer-readable storage medium of  claim 19 , further comprising instructions that cause the processor to:
 upsample a significance map from the base layer; and   use the upsampled significance map to decode one or more of the enhancement layers.   
     
     
         24 . The non-transitory computer-readable storage medium of  claim 19 , further comprising instructions that cause the processor to:
 reset a significance map from the base layer; and   decode coefficients only as zero coefficients or nonzero significant coefficients with no refinement coefficients in one or more of the enhancement layers.   
     
     
         25 . A method comprising:
 decoding a fine granularity scalability (FGS) base layer including base layer video blocks defining video information at a first spatial resolution;   decoding one or more FGS enhancement layers including enhancement layer video blocks defining video information at a second spatial resolution greater than the first spatial resolution; and   decoding each of the blocks in the FGS enhancement layers using a special DC mode in which a DC coefficient is predicted from a predetermined default value known to both an encoder and a decoder and AC coefficients are decoded without a prediction calculated from neighboring blocks, wherein the predetermined default value is determined before decoding the FGS base layer and the one or more FGS enhancement layers.   
     
     
         26 . The method of  claim 25 , wherein the FGS base layer defines the video information with a first quality level, and at least one of the FGS enhancement layers defines the video information with a second quality level greater than the first quality level. 
     
     
         27 . The method of  claim 25 , further comprising receiving a  1 -bit syntax element with at least one of the enhancement layers to signal the use of FGS coding for spatial scalability. 
     
     
         28 . A device comprising a decoder that decodes a fine granularity scalability (FGS) base layer including base layer video blocks defining video information at a first spatial resolution, encodes one or more FGS enhancement layers including enhancement layer video blocks defining video information at a second spatial resolution greater than the first spatial resolution, and decoding each of the blocks in the FGS enhancement layers using a special DC mode in which a DC coefficient is predicted from a predetermined default value known to both an encoder and a decoder and AC coefficients are decoded without a prediction calculated from neighboring blocks, wherein the predetermined default value is determined before the decoder decodes the FGS base layer and the one or more FGS enhancement layers. 
     
     
         29 . The device of  claim 28 , wherein the FGS base layer defines the video information with a first quality level, and at least one of the FGS enhancement layers defines the video information with a second quality level greater than the first quality level. 
     
     
         30 . The device of  claim 28 , further comprising receiving a  1 -bit syntax element with at least one of the enhancement layers to signal the use of FGS coding for spatial scalability. 
     
     
         31 . A device comprising:
 means for decoding a fine granularity scalability (FGS) base layer including base layer video blocks defining video information at a first spatial resolution;   means for decoding one or more FGS enhancement layers including enhancement layer video blocks defining video information at a second spatial resolution greater than the first spatial resolution; and   means for decoding each of the blocks in the FGS enhancement layers using a special DC mode in which a DC coefficient is predicted from a predetermined default value known to both an encoder and a decoder and AC coefficients are decoded without a prediction calculated from neighboring blocks, wherein the predetermined default value is determined before decoding the FGS base layer and the one or more FGS enhancement layers.   
     
     
         32 . The device of  claim 31 , wherein the FGS base layer defines the video information with a first quality level, and at least one of the FGS enhancement layers defines the video information with a second quality level greater than the first quality level. 
     
     
         33 . The device of  claim 31 , further comprising means for receiving a  1 -bit syntax element with at least one of the enhancement layers to signal the use of FGS coding for spatial scalability. 
     
     
         34 . A non-transitory computer-readable medium comprising instructions that, when executed, cause a processor to:
 decode a fine granularity scalability (FGS) base layer including base layer video blocks defining video information at a first spatial resolution;   decode one or more FGS enhancement layers including enhancement layer video blocks defining video information at a second spatial resolution greater than the first spatial resolution; and   decode each of the blocks in the FGS enhancement layers using a special DC mode in which a DC coefficient is predicted from a predetermined default value known to both an encoder and a decoder and AC coefficients are decoded without a prediction calculated from neighboring blocks, wherein the predetermined default value is determined before decoding the FGS base layer and the one or more FGS enhancement layers.   
     
     
         35 . The non-transitory computer-readable medium of  claim 34 , wherein the FGS base layer defines the video information with a first quality level, and at least one of the FGS enhancement layers defines the video information with a second quality level greater than the first quality level. 
     
     
         36 . The non-transitory computer-readable medium of  claim 34 , further comprising instructions that cause the processor to receive a  1 -bit syntax element with at least one of the enhancement layers to signal the use of FGS coding for spatial scalability.