US2018176577A1PendingUtilityA1
Content adaptive gain compensated prediction for next generation video coding
Est. expiryJan 30, 2033(~6.5 yrs left)· nominal 20-yr term from priority
Inventors:Atul Puri
H04N 19/63H04N 19/172H04N 19/96H04N 19/517H04N 19/52H04N 19/186H04N 19/119H04N 19/126H04N 19/61H04N 19/182H04N 19/124H04N 19/176H04N 19/14H04N 19/513H04N 19/577H04N 19/59H04N 19/167H04N 19/91H04N 19/12H04N 19/117H04N 19/82H04N 19/44H04N 19/159H04N 19/139H04N 19/31H04N 19/1887H04N 19/196H04N 19/105H04N 19/13H04N 19/50H04N 19/573H04N 19/593H04N 19/184H04N 19/17H04N 19/136H04N 19/192H04N 19/147H04N 19/122H04N 19/53H04N 19/40H04N 19/523
65
PatentIndex Score
0
Cited by
0
References
0
Claims
Abstract
Techniques related to content adaptive gain compensated prediction for next generation video coding are described.
Claims
exact text as granted — not AI-modified1 - 29 . (canceled)
30 . A computer-implemented method for video coding, comprising:
obtaining frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; selecting a partition pattern wherein each partition is associated with more than one pixel and among patterns that use a varying number or varying arrangement or both of partitions to form a frame; determining brightness gain compensation values for the reference frame by providing a gain value and an offset value for individual partitions; and applying locally adaptive gain compensation by adjusting the brightness of a partition of the reference frame and adjusted by the gain compensation value so that multiple gain compensation values are provided for a single frame and depend on the location of the partition within the frame.
31 . The method of claim 30 wherein each partition has an average gain value and an average offset value based on, at least in part, an average of pixel values in the partition.
32 . The method of claim 30 wherein the partition pattern is selected by using at least one of:
rate distortion optimization (RDO), and
a method that balances quality of image with bit cost for using a particular partition.
33 . The method of claim 30 wherein multiple partitions are tested to determine which partition results in a best balance of quality and bit cost, and based, at least in part, on distortion between a partition on the current frame and the corresponding partition on the reference frame, a bitrate associated with the gain and offset values, and the quantizer used for transform coefficients.
34 . The method of claim 30 comprising determining which partition results in a best balance of quality and bit cost, and based, at least in part, on minimizing J for:
J
=
D
+
λ
×
R
where
λ
=
c
×
(
Q
2
)
2
and where D is a distortion between a partition on the current frame and the corresponding partition on the reference frame, a bitrate R is associated with the gain and offset values, and λ is a langrangian multiplier where Q is the quantizer used for transform coefficients, and c is a constant found by heuristics and set at 0.7 as a middle value.
35 . The method of claim 30 wherein the partition pattern is selected from an index of predetermined partition patterns.
36 . The method of claim 35 wherein the index comprises index numbers each associated with a different pattern, and wherein some of the index numbers indicate the number of partitions in a corresponding pattern, and some index numbers indicate the size of the partitions in a corresponding pattern without expressly stating the number of partitions.
37 . The method of claim 30 wherein the partition pattern is selected separately from any partition formed for prediction blocks and for partitions formed for transform coding and quantization of chroma and luma values in the image data of the pixels.
38 . The method of claim 30 wherein the gain and offset values are quantized for transmission from an encoder and by using a predetermined index with a fixed number of intervals to maintain the transmission of the gain and offset values to a predetermined number of bits;
wherein a quantization index number is quantized rather than the actual gain or offset value;
wherein a pattern indicator, a gain indicator for each partition, and offset indicator for each partition are limited to seven, eight, and eight bits respectively to transmit the indication of the gain compensation for a frame from an encoder; and
wherein only the parameter index number and quantized gain value(s) and quantized offset value(s) are transmitted from an encoder to reconstruct the gain compensation.
39 . The method of claim 30 wherein total gain compensation is S t =a(S t-1 )+b, and wherein the gain a is calculated using at least one of:
(A) a moment method as:
a
=
(
E
(
S
t
2
(
i
,
j
)
)
-
(
E
(
S
t
(
i
,
j
)
)
)
2
(
E
(
S
t
-
k
2
(
i
,
j
)
)
-
(
E
(
S
t
-
k
(
i
,
j
)
)
)
2
wherein E is the expectation operator, S t is the brightness value of pixel (i, j) of the current frame t, S t-1 is the brightness value of a corresponding pixel (i, j) of the reference frame t−k that is selected to be used as a reference frame for the current frame, and
(B) a mean square error gradient minimization method as:
a
=
E
(
S
t
(
i
,
j
)
(
S
t
-
k
(
i
,
j
)
)
)
-
E
(
S
t
(
i
,
j
)
)
E
(
S
t
-
k
(
i
,
j
)
)
E
(
S
t
-
k
2
(
i
,
j
)
)
-
E
(
S
t
(
i
,
j
)
)
E
(
S
t
-
k
(
i
,
j
)
)
wherein E is the expectation operator, S t is the brightness value of pixel (i, j) of the current frame t, S t-1 is the brightness value of a corresponding pixel (i, j) of the reference frame t−k that is selected to be used as a reference frame for the current frame; and
wherein offset b is calculated as:
b=E ( st ( i,j ))− a×E ( st−k ( i,j ))
wherein E is the expectation operator, S t is the brightness value of pixel (i, j) of the current frame t, S t-1 is the brightness value of a corresponding pixel (i, j) of the reference frame t−k that is selected to be used as a reference frame for the current frame, and wherein a is the gain value between the two corresponding pixels.
40 . The method of claim 30 comprising:
calculating averaged block pixel values of individual blocks of a frame; and
adding up the block values of different blocks to determine sums for calculating the gain and offset values for different partitions.
41 . The method of claim 40 wherein the averaged block pixel values include at least one of: (1) the average of the pixel brightness values within the block, (2) the average of the square of the pixel brightness within the block, and (3) the average of the products (by multiplication) of corresponding pixel brightness values in the current and reference frames, and within the block.
42 . The method of claim 30 wherein the partitions are gain partitions defined independently of prediction partitions and coding partitions.
43 . The method of claim 30 wherein the current frame is an F-picture that provides the option to use a past frame, a future frame or both as a reference, and provides the option to use modified reference frames that are modified by morphing or synthesis wherein gain compensation is one or a plurality of morphing options.
44 . The method of claim 30 wherein each partition has an average gain value and an average offset value based on, at least in part, an average of pixel values in the partition;
wherein the partition pattern is selected by using at least one of:
rate distortion optimization (RDO), and
a method that balances quality of image with bit cost for using a particular partition;
wherein multiple partitions are tested to determine which partition results in a best balance of quality and bit cost, and based, at least in part, on distortion between a partition on the current frame and the corresponding partition on the reference frame, a bitrate associated with the gain and offset values, and the quantizer used for transform coefficients;
the method comprising determining which partition results in a best balance of quality and bit cost, and based, at least in part, on minimizing J for:
J
=
D
+
λ
×
R
where
λ
=
c
×
(
Q
2
)
2
and where D is a distortion between a partition on the current frame and the corresponding partition on the reference frame, a bitrate R is associated with the gain and offset values, and λ is a langrangian multiplier where Q is the quantizer used for transform coefficients, and c is a constant found by heuristics and set at 0.7 as a middle value;
wherein the partition pattern is selected from an index of predetermined partition patterns.
wherein the index comprises index numbers each associated with a different pattern, and wherein some of the index numbers indicate the number of partitions in a corresponding pattern, and some index numbers indicate the size of the partitions in a corresponding pattern without expressly stating the number of partitions;
wherein the partition pattern is selected separately from any partition formed for prediction blocks and for partitions formed for transform coding and quantization of chroma and luma values in the image data of the pixels;
wherein the gain and offset values are quantized for transmission from an encoder and by using a predetermined index with a fixed number of intervals to maintain the transmission of the gain and offset values to a predetermined number of bits;
wherein a quantization index number is quantized rather than the actual gain or offset value;
wherein a pattern indicator, a gain indicator for each partition, and offset indicator for each partition are limited to seven, eight, and eight bits respectively to transmit the indication of the gain compensation for a frame from an encoder;
wherein only the parameter index number and quantized gain value(s) and quantized offset value(s) are transmitted from an encoder to reconstruct the gain compensation;
the method comprising calculating motion compensation after the gain compensation is applied;
wherein total gain compensation is S t =a(S t-1 )+b, and wherein the gain a is calculated using at least one of:
(A) a moment method as:
a
=
(
E
(
S
t
2
(
i
,
j
)
)
-
(
E
(
S
t
(
i
,
j
)
)
)
2
(
E
(
S
t
-
k
2
(
i
,
j
)
)
-
(
E
(
S
t
-
k
(
i
,
j
)
)
)
2
wherein E is the expectation operator, S t is the brightness value of pixel (i, j) of the current frame t, S t-1 is the brightness value of a corresponding pixel (i, j) of the reference frame t−k that is selected to be used as a reference frame for the current frame, and
(B) a mean square error gradient minimization method as:
a
=
E
(
S
t
(
i
,
j
)
(
S
t
-
k
(
i
,
j
)
)
)
-
E
(
S
t
(
i
,
j
)
)
E
(
S
t
-
k
(
i
,
j
)
)
E
(
S
t
-
k
2
(
i
,
j
)
)
-
E
(
S
t
(
i
,
j
)
)
E
(
S
t
-
k
(
i
,
j
)
)
wherein offset b is calculated as:
b=E ( st ( i,j ))− a×E ( st−k ( i,j ))
wherein E is the expectation operator, S t is the brightness value of pixel (i, j) of the current frame t, S t-1 is the brightness value of a corresponding pixel (i, j) of the reference frame t−k that is selected to be used as a reference frame for the current frame, and wherein a is the gain value between the two corresponding pixels;
the method comprising calculating averaged block pixel values of individual blocks of a frame, and adding up the block values of different blocks to determine sums for calculating the gain and offset values for different partitions;
wherein the blocks are 4×4 pixels;
wherein the averaged block pixel values include at least one of: (1) the average of the pixel brightness values within the block, (2) the average of the square of the pixel brightness within the block, and (3) the average of the products (by multiplication) of corresponding pixel brightness values in the current and reference frames, and within the block; and
wherein the partitions are gain partitions defined independently of prediction partitions and coding partitions.
45 . A computer implemented method of video coding comprising:
performing compensation of inter-frame changes in brightness comprising:
calculating multiple sets of parameters wherein each set comprises a gain value and an offset value for a portion of a frame, and calculated by using brightness of pixels of a current frame and a reference frame;
transmitting the sets of parameters and an indication of which portion of the frame corresponds to which parameters to a decoder; and
using the parameters to form a modified reference frame that is lower in error than the original reference frame.
46 . A coder comprising:
an image buffer; and a graphics processing unit configured to:
obtain frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame;
select a partition pattern wherein each partition is associated with more than one pixel and among patterns that use a varying number or varying arrangement or both of partitions to form a frame;
determine brightness gain compensation values for the reference frame by providing a gain value and an offset value for individual partitions; and
apply locally adaptive gain compensation by adjusting the brightness of a partition of the reference frame and adjusted by the gain compensation value so that multiple gain compensation values are provided for a single frame and depend on the location of the partition within the frame.
47 . The coder of claim 46 wherein each partition has an average gain value and an average offset value based on, at least in part, an average of pixel values in the partition;
wherein the partition pattern is selected by using at least one of:
rate distortion optimization (RDO), and
a method that balances quality of image with bit cost for using a particular partition;
wherein multiple partitions are tested to determine which partition results in a best balance of quality and bit cost, and based, at least in part, on distortion between a partition on the current frame and the corresponding partition on the reference frame, a bitrate associated with the gain and offset values, and the quantizer used for transform coefficients;
the graphics processing unit being configured to determine which partition results in a best balance of quality and bit cost, and based, at least in part, on minimizing J for:
J
=
D
+
λ
×
R
where
λ
=
c
×
(
Q
2
)
2
and where D is a distortion between a partition on the current frame and the corresponding partition on the reference frame, a bitrate R is associated with the gain and offset values, and λ is a langrangian multiplier where Q is the quantizer used for transform coefficients, and c is a constant found by heuristics and set at 0.7 as a middle value;
wherein the partition pattern is selected from an index of predetermined partition patterns.
wherein the index comprises index numbers each associated with a different pattern, and wherein some of the index numbers indicate the number of partitions in a corresponding pattern, and some index numbers indicate the size of the partitions in a corresponding pattern without expressly stating the number of partitions;
wherein the partition pattern is selected separately from any partition formed for prediction blocks and for partitions formed for transform coding and quantization of chroma and luma values in the image data of the pixels;
wherein the gain and offset values are quantized for transmission from an encoder and by using a predetermined index with a fixed number of intervals to maintain the transmission of the gain and offset values to a predetermined number of bits;
wherein a quantization index number is quantized rather than the actual gain or offset value;
wherein a pattern indicator, a gain indicator for each partition, and offset indicator for each partition are limited to seven, eight, and eight bits respectively to transmit the indication of the gain compensation for a frame from an encoder;
wherein only the parameter index number and quantized gain value(s) and quantized offset value(s) are transmitted from an encoder to reconstruct the gain compensation;
the graphics processing unit being configured to calculate motion compensation after the gain compensation is applied;
wherein total gain compensation is S t =a(S t-1 )+b, and wherein the gain a is calculated using at least one of:
(A) a moment method as:
a
=
(
E
(
S
t
2
(
i
,
j
)
)
-
(
E
(
S
t
(
i
,
j
)
)
)
2
(
E
(
S
t
-
k
2
(
i
,
j
)
)
-
(
E
(
S
t
-
k
(
i
,
j
)
)
)
2
wherein E is the expectation operator, S t is the brightness value of pixel (i,j) of the current frame t, S t-1 is the brightness value of a corresponding pixel (i,j) of the reference frame t−k that is selected to be used as a reference frame for the current frame, and
(B) a mean square error gradient minimization method as:
a
=
E
(
S
t
(
i
,
j
)
(
S
t
-
k
(
i
,
j
)
)
)
-
E
(
S
t
(
i
,
j
)
)
E
(
S
t
-
k
(
i
,
j
)
)
E
(
S
t
-
k
2
(
i
,
j
)
)
-
E
(
S
t
(
i
,
j
)
)
E
(
S
t
-
k
(
i
,
j
)
)
wherein offset b is calculated as:
b=E ( st ( i,j ))− a×E ( st−k ( i,j ))
wherein E is the expectation operator, S t is the brightness value of pixel (i, j) of the current frame t, S t-1 is the brightness value of a corresponding pixel (i, j) of the reference frame t−k that is selected to be used as a reference frame for the current frame, and wherein a is the gain value between the two corresponding pixels;
the graphics processing unit being configured to calculate averaged block pixel values of individual blocks of a frame, and add up the block values of different blocks to determine sums for calculating the gain and offset values for different partitions;
wherein the blocks are 4×4 pixels;
wherein the averaged block pixel values include at least one of: (1) the average of the pixel brightness values within the block, (2) the average of the square of the pixel brightness within the block, and (3) the average of the products (by multiplication) of corresponding pixel brightness values in the current and reference frames, and within the block; and
wherein the partitions are gain partitions defined independently of prediction partitions and coding partitions.
48 . At least one computer readable memory comprising instructions, that when executed by a computing device, cause the computing device to:
obtain frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; select a partition pattern wherein each partition is associated with more than one pixel and among patterns that use a varying number or varying arrangement or both of partitions to form a frame; and determine brightness gain compensation values for the reference frame and by providing a gain value and an offset value for individual partitions; and apply locally adaptive gain compensation by adjusting the brightness of a partition of the reference frame and adjusted by the gain compensation value so that multiple gain compensation values are provided for a single frame and depend on the location of the partition within the frame.
49 . The computer readable medium of claim 48 wherein each partition has an average gain value and an average offset value based on, at least in part, an average of pixel values in the partition;
wherein the partition pattern is selected by using at least one of:
rate distortion optimization (RDO), and
a method that balances quality of image with bit cost for using a particular partition;
wherein multiple partitions are tested to determine which partition results in a best balance of quality and bit cost, and based, at least in part, on distortion between a partition on the current frame and the corresponding partition on the reference frame, a bitrate associated with the gain and offset values, and the quantizer used for transform coefficients;
the instructions causing the computing device to determine which partition results in a best balance of quality and bit cost, and based, at least in part, on minimizing J for:
J
=
D
+
λ
×
R
where
λ
=
c
×
(
Q
2
)
2
and where D is a distortion between a partition on the current frame and the corresponding partition on the reference frame, a bitrate R is associated with the gain and offset values, and λ is a langrangian multiplier where Q is the quantizer used for transform coefficients, and c is a constant found by heuristics and set at 0.7 as a middle value;
wherein the partition pattern is selected from an index of predetermined partition patterns.
wherein the index comprises index numbers each associated with a different pattern, and wherein some of the index numbers indicate the number of partitions in a corresponding pattern, and some index numbers indicate the size of the partitions in a corresponding pattern without expressly stating the number of partitions;
wherein the partition pattern is selected separately from any partition formed for prediction blocks and for partitions formed for transform coding and quantization of chroma and luma values in the image data of the pixels;
wherein the gain and offset values are quantized for transmission from an encoder and by using a predetermined index with a fixed number of intervals to maintain the transmission of the gain and offset values to a predetermined number of bits;
wherein a quantization index number is quantized rather than the actual gain or offset value;
wherein a pattern indicator, a gain indicator for each partition, and offset indicator for each partition are limited to seven, eight, and eight bits respectively to transmit the indication of the gain compensation for a frame from an encoder;
wherein only the parameter index number and quantized gain value(s) and quantized offset value(s) are transmitted from an encoder to reconstruct the gain compensation;
the instructions causing the computing device to calculate motion compensation after the gain compensation is applied;
wherein total gain compensation is S t =a(S t-1 )+b, and wherein the gain a is calculated using at least one of:
(A) a moment method as:
a
=
(
E
(
S
t
2
(
i
,
j
)
)
-
(
E
(
S
t
(
i
,
j
)
)
)
2
(
E
(
S
t
-
k
2
(
i
,
j
)
)
-
(
E
(
S
t
-
k
(
i
,
j
)
)
)
2
,
and
(B) a mean square error gradient minimization method as:
a
=
E
(
S
t
(
i
,
j
)
(
S
t
-
k
(
i
,
j
)
)
)
-
E
(
S
t
(
i
,
j
)
)
E
(
S
t
-
k
(
i
,
j
)
)
E
(
S
t
-
k
2
(
i
,
j
)
)
-
E
(
S
t
(
i
,
j
)
)
E
(
S
t
-
k
(
i
,
j
)
)
wherein offset b is calculated as:
b=E ( st ( i,j ))− a×E ( st−k ( i,j ))
wherein E is the expectation operator, S t is the brightness value of pixel (i, j) of the current frame t, S t-1 is the brightness value of a corresponding pixel (i, j) of the reference frame t−k that is selected to be used as a reference frame for the current frame, and wherein a is the gain value between the two corresponding pixels;
the instructions causing the computing device to calculate averaged block pixel values of individual blocks of a frame, and add up the block values of different blocks to determine sums for calculating the gain and offset values for different partitions;
wherein the blocks are 4×4 pixels;
wherein the averaged block pixel values include at least one of: (1) the average of the pixel brightness values within the block, (2) the average of the square of the pixel brightness within the block, and (3) the average of the products (by multiplication) of corresponding pixel brightness values in the current and reference frames, and within the block; and
wherein the partitions are gain partitions defined independently of prediction partitions and coding partitions.Cited by (0)
No later patents cite this yet.
References (0)
No backward citations on record.