US2017013279A1PendingUtilityA1
Content adaptive dominant motion compensated prediction for next generation video coding
Est. expiryNov 13, 2032(~6.3 yrs left)· nominal 20-yr term from priority
H04N 19/573H04N 19/147H04N 19/85H04N 19/46H04N 19/136H04N 19/122H04N 19/119H04N 19/176H04N 19/82H04N 19/44H04N 19/139H04N 19/527H04N 19/105H04N 19/12H04N 19/172H04N 19/61H04N 19/513H04N 19/91
63
PatentIndex Score
0
Cited by
0
References
0
Claims
Abstract
Techniques related to dominant motion compensated prediction for next generation video coding are described.
Claims
exact text as granted — not AI-modified1 - 47 . (canceled)
48 . A computer-implemented method for video coding, comprising:
obtaining frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; forming a warped global compensated reference frame by displacing at least one portion of the decoded reference frame by using global motion trajectories; determining a motion vector indicating the motion of the at least one portion and motion from a position based on the warped global compensated reference frame to a position at the current frame; and forming a prediction portion based, at least in part, on the motion vectors and corresponding to a portion on the current frame.
49 . The method of claim 48 wherein the at least one portion is a block of pixels used as a unit to divide the current frame and the reference frame into a plurality of the blocks.
50 . The method of claim 48 wherein the at least one portion is at least one tile of pixels, each tile being at least 64×64 pixels, and used as a unit to divide the current frame and the reference frame into a plurality of the tiles;
the method comprising grouping tiles together based on common association with an object in the frame to form the at least one portion; and forming a single motion vector for each group of tiles; and grouping the tiles based on a merge map transmittable from an encoder to a decoder.
51 . The method of claim 48 wherein the at least one portion is a region of pixels shaped and sized depending on an object associated with the region; and
wherein a boundary of the region is at least one of:
a shape that resembles the shape of the object associated with the region, and
a rectangle placed around the object associated with the region.
52 . The method of claim 48 wherein the region is associated with at least one of:
a background of the frame,
a foreground of the frame, and
a moving object in the frame; and
wherein each region has a single motion vector.
53 . The method of claim 48 wherein forming a warped global compensated reference frame comprises using the global motion trajectories at the outer corners of the frame; and using an affine or perspective global motion compensation method.
54 . The method of claim 48 comprising:
performing dominant motion compensation comprising locally applied global motion compensation so that at least one other set of global motion trajectories are used at corners of at least one region on the frame that is less than the entire frame to form a displaced region;
using the pixel values of the displaced region to form a prediction region that corresponds to a region on the current frame; and
providing the option on the at least one region on a region-by-region basis to select a prediction formed by:
(1) a motion vector to form a prediction for the at least one region and using global motion compensation applied to the entire frame, or
(2) applying local global motion compensation with a set of global motion trajectories at the region and using displaced pixel values of the region to form a prediction.
55 . The method of claim 48 comprising applying local global motion compensation with a set of global motion trajectories applied at a region of the reference frame that has an area less than the entire reference frame, and using motion vectors to form a prediction for the at least one region.
56 . The method of claim 48 comprising providing the option to select a mode for a frame among:
(1) use the dominant motion compensated reference frame prediction,
(2) use blended prediction of multiple dominant motion compensated reference frames,
(3) use dominant motion compensated reference with differential translational motion vector for prediction, and
(4) use dominant motion compensated reference with differential translational motion vector for prediction, blended with another reference frame.
57 . The method of claim 48 comprising performing motion compensated morphed reference prediction using bilinear interpolation and motion compensation (MC) filter to form a morphed reference frame MRef, tPred h as the intermediate horizontal interpolation, and pred ji as the final motion compensated morphed reference prediction:
MRef
[
i
′
]
[
j
′
]
=
(
(
8
-
p
x
)
(
8
-
p
y
)
Ref
[
y
0
]
[
x
0
]
+
p
x
(
8
-
p
y
)
Ref
[
y
0
]
[
x
0
+
1
]
+
p
y
(
8
-
p
x
)
Ref
[
y
0
+
1
]
[
x
0
]
+
p
y
p
x
Ref
[
y
0
+
1
]
[
x
0
+
1
]
+
31
)
6
tPred
h
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
j
]
[
k
]
·
MRef
[
i
′
+
m
]
[
j
′
+
n
+
k
-
N
t
2
+
1
]
where
m
=
[
-
N
t
/
2
+
1
,
H
b
+
N
t
/
2
-
1
]
,
n
=
[
0
,
W
b
-
1
]
,
Pred
ji
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
i
]
[
k
]
·
tPred
h
[
m
+
k
-
N
t
2
+
1
]
[
n
]
where
m
=
[
0
,
H
b
-
1
]
,
n
=
[
0
,
W
b
-
1
]
,
and where:
(iMVx, iMVy) is the transmitted motion vector in Sub-Pel Unit (f s ) for a block at (j, i) of size (W b ×H b ); A, B, C, D, E, & F are affine parameters calculated from the three Motion trajectories transmitted; using separable motion compensation (MC) Filters with filter coefficients h[f s ][N t ] of norm T, f s is the Sub-Pel Factor (e.g. 2=Half Pel, 4=Quarter Pel, 8=Eighth Pel), where N t is the number MC Filter Taps, and
i′=i +( iMVy/f s )
j′=j +( iMVx/f s )
p i =iMVy & ( f s −1)
p j =iMVx & ( f s −1)
(j′, i′) is integer motion adjusted current pixel location in Morphed Reference Image; p j , p i are the ⅛ th pel phases in the Morphed Reference Image;
x =( A*j′+B*i′+C<<r )>> r
y =( D*j′+E*i′+F<<s )>> s
where (x, y) is the reference pixel coordinate in ⅛ th Pel accuracy for location (j′, i′)
p y =y & 0×7
p x =x & 0×7
y 0 =>>3
x 0 =x>> 3
where (x 0 , y 0 ) is the integer pel location in Ref Image; p x , p y is the ⅛ th pel phase;
M Ref[ i′][j ′]=((8− p x )*(8− p y )*Ref[ y 0 ][x 0 ]+p x *(8− p y )*Ref[ y 0 ][x 0 +1]+ p y *(8− p x )*Ref[ y 0 +1][ x 0 ]+p y *p x *Ref[ y 0 +1][ x 0 +1]+31)>>6
t Pred h [m][n ]=SUM k ( h[p j ][k]*M Ref[ i′+m][j′+n+k ])/ T,
where m=[−N t /2−1, H b +N t /2], where n=[0, W b −1], where k=[−N t /2−1, N t /2],
Pred ji [m ]=SUM k [p j ][k]*t Pred h [m+k][n ])/ T,
where m=[0, H b −1], where n=[0, W b −1], where k=[−N t /2−1, +N t /2].
58 . The method of claim 48 comprising performing morphed reference prediction using block motion compensation (MC) filtering to form a morphed reference frame Mref, and Predh as the intermediate horizontal interpolation:
tPred
h
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
x
]
[
k
]
·
Ref
[
y
0
+
m
]
[
x
0
+
n
+
k
-
N
t
2
+
1
]
m
=
[
-
N
t
/
2
+
1
,
H
s
+
N
t
/
2
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
MRef
[
i
+
m
]
[
j
+
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
y
]
[
k
]
·
tPred
h
[
m
+
k
-
N
t
2
+
1
]
[
n
]
where
m
=
[
0
,
H
s
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
and where A, B, C, D, E, & F are affine parameters calculated from the three Motion trajectories transmitted; using separable MC Filters with filter coefficients h[fs][N t ] of norm T; fs is the Sub-Pel Factor (e.g. 2=Half Pel, 4=Quarter Pel, 8=Eighth Pel), and where N t is the number MC Filter Taps
x =( A*j+B*i+C<<r )>> r
y =( D*j+E*i+F<<s )>> s
(j, i) is every (W s ×H s ) sub-block location in current image, x and y are reference pixel coordinates in ⅛th Pel accuracy;
p y =y & 0×7
p x =x & 0×7
y 0 =y >> 3
x 0 =x>> 3
(x 0 , y 0 ) is the integer pel location in the reference frame (Ref Image); p x , p y is the ⅛th pel phase.
t Pred h [m][n ]=SUM k ( h[p x ][k ]*Ref[ y 0 +m][x 0 +n+k ])/ T,
m=[−N t /2−1, H s +N t /2], n=[0, W s −1], k=[−N t /2−1, +N t /2]; and
M Ref[ i+m][j+n ]=SUM k ( h[p y ][k]*t Pred h [m+k][n ])/ T,
m=[0, H s −1], n=[0, W s −1], k=[−N t /2−1, +N t /2].
59 . The method of claim 48 comprising performing motion compensated morphed reference prediction using single loop motion compensation (MC) filtering to form a morphed reference (Mref) and predictions tPred h as the intermediate horizontal interpolation, and Pred ji as the final motion compensated morphed reference prediction for block of size W b ×H b at (j, i):
tPred
h
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
x
]
[
k
]
·
Ref
[
y
0
+
m
]
[
x
0
+
n
+
k
-
N
t
2
+
1
]
for
:
m
=
[
-
N
t
/
2
+
1
,
H
s
+
N
t
/
2
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
Pred
ji
[
uH
s
+
m
]
[
vW
s
+
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
y
]
[
k
]
·
tPred
h
[
m
+
k
-
N
t
2
+
1
]
[
n
]
for
:
m
=
[
0
,
H
s
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
u
=
[
0
,
H
b
/
H
s
-
1
]
,
v
=
[
0
,
W
b
/
W
s
-
1
]
,
and where:
(iMVx, iMVy) is the transmitted Motion Vector in Sub-Pel Units (fs) for a block at (j, i) of size (W b ×H b ); A, B, C, D, E, & F are affine parameters calculated from the three Motion trajectories transmitted; using separable MC Filters with filter coefficients h[fs][N t ] of norm T, fs is the Sub-Pel Factor (e.g. 2=Half Pel, 4=Quarter Pel, 8=Eighth Pel), and N t is the number MC Filter Taps;
i ′=( i+u*H s )* fs+iMVx
j ′=( j+v*W s )* fs+iMVy
where (j, i) is the current block pixel location, (u, v) is the index of every (W s ×H s ) sub-block within given current block of (W b ×H b ), and (W s ×H s ) sub-block; Below, i′, j′ is motion adjusted current pixel location in fs sub-pel accuracy,
x =(( A*j′+B*i ′+( C*fs )<< r )>>( r+ 3)
y =(( D*j′+E*i ′+( F*fs )<< s )>>( s+ 3)
where x & y are reference pixel coordinates in fs sub-pel accuracy
p y =y & ( fs− 1)
p x =x & ( fs− 1)
y 0 =y/fs
x 0 =x/fs
where y 0 , x 0 is the integer pel location in Ref Image, p x , p y is the ⅛th pel phase;
t Pred h [m][n ]=SUM k ( h[p x ][k ]*Ref[ y 0 +m][x 0 +n+k ])/ T,
m=[−N t /2−1, H s +N t /2],
n=[ 0, W s −1],
k=[−N t /2−1,+ N t /2],
Pred ji [u*H s +m][v*W s +n ]=SUM k ( h[p y ][k]*t Pred h [m+k][n ])/ T,
m=[ 0, H s −1],
n=[ 0, W s −1],
k=[−N t /2−1,+ N t /2],
v=[ 0, W b /W s −1],
u=[ 0, H b /H s −1].
60 . The method of claim 48 wherein the at least one portion is at least one of:
(1) a block of pixels used as a unit to divide the current frame and the reference frame into a plurality of the blocks;
(2) at least one tile of pixels, each tile being at least 64×64 pixels, and used as a unit to divide the current frame and the reference frame into a plurality of the tiles;
the method comprising at least one of:
grouping tiles together based on common association with an object in the frame to form the at least one portion; and forming a single motion vector for each group of tiles,
grouping the tiles based on a merge map transmittable from an encoder to a decoder;
(3) a region of pixels shaped and sized depending on an object associated with the region, wherein a boundary of the region is at least one of:
a shape that resembles the shape of the object associated with the region, and
a rectangle placed around the object associated with the region;
wherein the region is associated with at least one of:
a background of the frame,
a foreground of the frame, and
a moving object in the frame;
the method comprising defining the region based on a boundary map transmittable from an encoder to a decoder;
wherein forming a warped global compensated reference frame comprises using the global motion trajectories at the outer corners of the frame;
wherein forming a warped global compensated reference frame comprises using an affine or perspective global motion compensation method;
wherein the at least one portion comprises a frame divided into a background and a foreground, and wherein determining motion vectors comprises providing the background and foreground each with one motion vector;
the method comprising performing dominant motion compensation comprising locally applied global motion compensation so that at least one other set of global motion trajectories are used at corners of at least one region on the frame that is less than the entire frame to form a displaced region; and using the pixel values of the displaced region to form a prediction region that corresponds to a region on the current frame;
the method comprising at least one of:
performing local global motion compensation on multiple regions of the frame by using a different set of global motion trajectories on each region;
wherein each region is a tile, and dividing the frame into the tiles, and wherein each tile has a set of global motion trajectories;
providing the option to perform local global motion compensation on a fraction of a tile in addition to entire tiles;
wherein each region is shaped and sized depending on an object associated with the region;
wherein the object is one of: a foreground, a background, and an object moving in the frame;
the method comprising providing the option on the at least one region on a region-by-region basis to select a prediction formed by:
(1) a motion vector to form a prediction for the at least one region and using global motion compensation applied to the entire frame, or
(2) applying local global motion compensation with a set of global motion trajectories at the region and using displaced pixel values of the region to form a prediction;
the method comprising applying local global motion compensation with a set of global motion trajectories applied at a region of the reference frame that has an area less than the entire reference frame, and using motion vectors to form a prediction for the at least one region;
the method comprising providing the option to select a mode for a frame among:
(1) use the dominant motion compensated reference frame prediction,
(2) use blended prediction of multiple dominant motion compensated reference frames,
(3) use dominant motion compensated reference with differential translational motion vector for prediction, and
(4) use dominant motion compensated reference with differential translational motion vector for prediction, blended with another reference frame;
the method comprising at least one of (a) to (c):
(a) performing motion compensated morphed reference prediction using bilinear interpolation and motion compensation (MC) filter to form a morphed reference frame MRef, tPred h as the intermediate horizontal interpolation, and pred ji as the final motion compensated morphed reference prediction:
MRef
[
i
′
]
[
j
′
]
=
(
(
8
-
p
x
)
(
8
-
p
y
)
Ref
[
y
0
]
[
x
0
]
+
p
x
(
8
-
p
y
)
Ref
[
y
0
]
[
x
0
+
1
]
+
p
y
(
8
-
p
x
)
Ref
[
y
0
+
1
]
[
x
0
]
+
p
y
p
x
Ref
[
y
0
+
1
]
[
x
0
+
1
]
+
31
)
6
tPred
h
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
j
]
[
k
]
·
MRef
[
i
′
+
m
]
[
j
′
+
n
+
k
-
N
t
2
+
1
]
where
m
=
[
-
N
t
/
2
+
1
,
H
b
+
N
t
/
2
-
1
]
,
n
=
[
0
,
W
b
-
1
]
,
Pred
ji
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
i
]
[
k
]
·
tPred
h
[
m
+
k
-
N
t
2
+
1
]
[
n
]
where
m
=
[
0
,
H
b
-
1
]
,
n
=
[
0
,
W
b
-
1
]
,
and where:
(iMVx, iMVy) is the transmitted motion vector in Sub-Pel Unit (f s ) for a block at (j, i) of size (W b ×H b ); A, B, C, D, E, & F are affine parameters calculated from the three Motion trajectories transmitted; using separable motion compensation (MC) Filters with filter coefficients h[f s ][N t ] of norm T, fs is the Sub-Pel Factor (e.g. 2=Half Pel, 4=Quarter Pel, 8=Eighth Pel), where N t is the number MC Filter Taps, and
i′=i +( iMVy/f s )
j′=j +( iMVx/f s )
p i =iMVy & ( f s −1)
p j =iMVx & ( f s −1)
(j′, i′) is integer motion adjusted current pixel location in Morphed Reference Image; p j , p i are the ⅛ th pel phases in the Morphed Reference Image;
x =( A*j′+B*i′+C<<r )>> r
y =( D*j′+E*i′+F<<s )>> s
where (x, y) is the reference pixel coordinate in ⅛ th Pel accuracy for location (j′, i′)
p y =y & 0×7
p x =x & 0×7
y 0 =>>3
x 0 =x>> 3
where (x 0 , y 0 ) is the integer pel location in Ref Image; p x , p y is the ⅛ th pel phase;
M Ref[ i′][j ′]=((8− p x )*(8− p y )*Ref[ y 0 ][x 0 ]+p x *(8− p y )*Ref[ y 0 ][x 0 +1]+ p y *(8− p x )*Ref[ y 0 +1][ x 0 ]+p y *p x *Ref[ y 0 +1][ x 0 +1]+31)>>6
t Pred h [m][n ]=SUM k ( h[p j ][k]*M Ref[ i′+m][j′+n+k ])/ T,
where m=[−N t /2−1, H b +N t /2], where n=[0, W b −1], where k=[−N t /2−1, N t /2],
Pred ji [m][n ]=SUM k ( h[p j ][k]*t Pred h [m+k][n ])/ T,
where m=[0, H b −1], where n=[0, W b −1], where k=[−N t /2−1, +N t /2];
(b) performing morphed reference prediction using block motion compensation (MC) filtering to form a morphed reference frame Mref, and Predh as the intermediate horizontal interpolation:
tPred
h
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
x
]
[
k
]
·
Ref
[
y
0
+
m
]
[
x
0
+
n
+
k
-
N
t
2
+
1
]
m
=
[
-
N
t
/
2
+
1
,
H
s
+
N
t
/
2
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
MRef
[
i
+
m
]
[
j
+
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
y
]
[
k
]
·
tPred
h
[
m
+
k
-
N
t
2
+
1
]
[
n
]
where
m
=
[
0
,
H
s
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
and where A, B, C, D, E, & F are affine parameters calculated from the three Motion trajectories transmitted; using separable MC Filters with filter coefficients h[fs][N t ] of norm T; fs is the Sub-Pel Factor (e.g. 2=Half Pel, 4=Quarter Pel, 8=Eighth Pel), and where N t is the number MC Filter Taps
x =( A*j+B*i+C<<r )>> r
y =( D*j+E*i+F<<s )>> s
(j, i) is every (W s ×H s ) sub-block location in current image, x and y are reference pixel coordinates in ⅛th Pel accuracy;
p y =y & 0×7
p x =x & 0×7
y 0 =y>> 3
x 0 =x>> 3
(x 0 , y 0 ) is the integer pel location in the reference frame (Ref Image); p x , p y is the ⅛th pel phase;
t Pred h [m][n ]=SUM k ( h[p x ][k ]*Ref[ y 0 +m][x 0 +n+k ])/ T,
m=[−N t /2−1, H s +N t /2], n=[0, W s −1], k=[−N t /2−1, +N t /2];
and
M Ref[ i+m][j+n ]=SUM k ( h[p y ][k]*t Pred h [m+k][n ])/ T,
m=[0, H s −1], n=[0, W s −1], k=[−N t /2−1, +N t /2]; and
(c) performing motion compensated morphed reference prediction using single loop motion compensation (MC) filtering to form a morphed reference (Mref) and predictions tPred h as the intermediate horizontal interpolation, and Pred ji as the final motion compensated morphed reference prediction for block of size W b ×H b at (j, i):
tPred
h
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
x
]
[
k
]
·
Ref
[
y
0
+
m
]
[
x
0
+
n
+
k
-
N
t
2
+
1
]
for
:
m
=
[
-
N
t
/
2
+
1
,
H
s
+
N
t
/
2
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
Pred
ji
[
uH
s
+
m
]
[
vW
s
+
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
y
]
[
k
]
·
tPred
h
[
m
+
k
-
N
t
2
+
1
]
[
n
]
for
:
m
=
[
0
,
H
s
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
u
=
[
0
,
H
b
/
H
s
-
1
]
,
v
=
[
0
,
W
b
/
W
s
-
1
]
,
and where:
(iMVx, iMVy) is the transmitted Motion Vector in Sub-Pel Units (fs) for a block at (j, i) of size (W b ×H b ); A, B, C, D, E, & F are affine parameters calculated from the three Motion trajectories transmitted; using separable MC Filters with filter coefficients h[fs][N t ] of norm T, fs is the Sub-Pel Factor (e.g. 2=Half Pel, 4=Quarter Pel, 8=Eighth Pel), and N t is the number MC Filter Taps;
i ′=( i+u*H s )* fs+iMVx
j ′=( j+v*W s )* fs+iMVy
where (j, i) is the current block pixel location, (u, v) is the index of every (W s ×H s ) sub-block within given current block of (W b ×H b ), and (W s ×H s ) sub-block; Below, i′, j′ is motion adjusted current pixel location in fs sub-pel accuracy,
x =(( A*j′+B*i ′+( C*fs )<< r )>>( r+ 3)
y =(( D*j′+E*i ′+( F*fs )<< s )>>( s+ 3)
where x & y are reference pixel coordinates in fs sub-pel accuracy
p y =y & ( fs− 1)
p x =x & ( fs− 1)
y 0 =y/fs
x 0 =x/fs
where y 0 , x 0 is the integer pel location in Ref Image, p x , p y is the ⅛th pel phase;
t Pred h [m][n ]=SUM k ( h[p x ][k ]*Ref[ y 0 +m][x 0 +n+k ])/ T,
m=[−N t /2−1, H s +N t /2],
n=[ 0, W s −1],
k=[−N t /2−1,+ N t /2],
Pred ji [u*H s +m][v*W s +n ]=SUM k ( h[p y ][k]*t Pred h [m+k][n ])/ T,
m=[ 0, H s −1],
n=[ 0, W s −1],
k=[−N t /2−1,+ N t /2],
v=[ 0, W b /W s −1],
u=[ 0, H b /H s −1].
61 . A computer-implemented method for video coding, comprising:
obtaining frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; dividing the reference frame into a plurality of portions that are less than the area of the entire frame; performing dominant motion compensation comprising applying local global motion compensation on at least one of the portions by displacing the at least one portion of the decoded reference frame by using global motion trajectories at a boundary of the portion; and forming a prediction portion that corresponds to a portion on the current frame, and by using the pixel values of the displaced portion.
62 . The method of claim 61 comprising performing local global motion compensation on a plurality of the portions by using a different set of global motion trajectories on each portion of the plurality of portions.
63 . The method of claim 62 wherein each portion is a tile, the method comprising dividing the frame into the tiles, and wherein each tile has a set of global motion trajectories.
64 . The method of claim 63 comprising grouping a plurality of the tiles into a region, and applying the same global motion trajectories on the tiles within the same region, and different sets of global motion trajectories depending on the region.
65 . The method of claim 48 comprising grouping a plurality of the portions into a region, and applying the same global motion trajectories on the portions within the same region, and different sets of global motion trajectories depending on the region; and wherein at least one of:
each portion is shaped and sized depending on an object associated with the portion,
the object is one of: a foreground, a background, and an object moving in the frame, and
the portion is a rectangle placed about the object.
66 . The method of claim 61 comprising forming a portion of the background of the reference frame, and a portion of the foreground of the reference frame each with a different set of local global motion trajectories for each portion.
67 . The method of claim 61 comprising performing local global motion compensation on a plurality of the portions by using a different set of global motion trajectories on each portion of the plurality of portions;
wherein each portion is a tile, the method comprising dividing the frame into the tiles, and wherein each tile has a set of global motion trajectories;
the method comprising providing the option to perform local global motion compensation on a fraction of a tile in addition to entire tiles;
wherein local global motion compensation trajectories are provided to half-tiles or quarter-tiles;
the method comprising at least one of:
grouping a plurality of the tiles into a region, and applying the same global motion trajectories on the tiles within the same region, and different sets of global motion trajectories depending on the region, and
grouping a plurality of the portions into a region, and applying the same global motion trajectories on the portions within the same region, and different sets of global motion trajectories depending on the region;
wherein each portion is shaped and sized depending on an object associated with the portion;
wherein the object is one of: a foreground, a background, and an object moving in the frame;
wherein the portion is a rectangle placed about the object;
the method comprising forming a portion of the background of the reference frame, and a portion of the foreground of the reference frame each with a different set of local global motion trajectories for each portion.
68 . A coder comprising:
an image buffer; and a graphics processing unit configured to:
obtain frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame;
divide the reference frame into a plurality of portions that are less than the area of the entire frame;
perform dominant motion compensation comprising applying local global motion compensation on at least one of the portions by displacing the at least one portion of the decoded reference frame by using global motion trajectories at a boundary of the portion; and
form a prediction portion that corresponds to a portion on the current frame and by using the pixel values of the displaced portion.
69 . The coder of claim 68 the graphics processing unit configured to:
perform local global motion compensation on a plurality of the portions by using a different set of global motion trajectories on each portion of the plurality of portions;
wherein each portion is a tile, the graphics processing unit configured to divide the frame into the tiles, and wherein each tile has a set of global motion trajectories;
the graphics processing unit configured to provide the option to perform local global motion compensation on a fraction of a tile in addition to entire tiles;
wherein local global motion compensation trajectories are provided to half-tiles or quarter-tiles;
the graphics processing unit configured to at least one of:
group a plurality of the tiles into a region, and apply the same global motion trajectories on the tiles within the same region, and different sets of global motion trajectories depending on the region; and
group a plurality of the portions into a region, and apply the same global motion trajectories on the portions within the same region, and different sets of global motion trajectories depending on the region;
wherein each portion is shaped and sized depending on an object associated with the portion;
wherein the object is one of: a foreground, a background, and an object moving in the frame;
wherein the portion is a rectangle placed about the object;
the graphics processing unit configured to form a portion of the background of the reference frame, and a portion of the foreground of the reference frame each with a different set of local global motion trajectories for each portion.
70 . A coder comprising:
an image buffer; and a graphics processing unit configured to:
obtain frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame;
form a warped global compensated reference frame by displacing at least one portion of the decoded reference frame by using global motion trajectories;
determine a motion vector indicating the motion of the at least one portion and motion from a position based on the warped global compensated reference frame to a position at the current frame; and
form a prediction portion based, at least in part, on the motion vectors and corresponding to a portion on the current frame.
71 . The coder of claim 70 wherein the at least one portion is at least one of:
(1) a block of pixels used as a unit to divide the current frame and the reference frame into a plurality of the blocks;
(2) at least one tile of pixels, each tile being at least 64×64 pixels, and used as a unit to divide the current frame and the reference frame into a plurality of the tiles;
the graphics processing unit being configured to at least one of:
group tiles together based on common association with an object in the frame to form the at least one portion; and form a single motion vector for each group of tiles,
group the tiles based on a merge map transmittable from an encoder to a decoder;
(3) a region of pixels shaped and sized depending on an object associated with the region, wherein a boundary of the region is at least one of:
a shape that resembles the shape of the object associated with the region, and
a rectangle placed around the object associated with the region;
wherein the region is associated with at least one of:
a background of the frame,
a foreground of the frame, and
a moving object in the frame;
the graphics processing unit being configured to define the region based on a boundary map transmittable from an encoder to a decoder;
wherein form a warped global compensated reference frame comprises using the global motion trajectories at the outer corners of the frame;
wherein form a warped global compensated reference frame comprises using an affine or perspective global motion compensation method;
wherein the at least one portion comprises a frame divided into a background and a foreground, and wherein determining motion vectors comprises providing the background and foreground each with one motion vector;
the graphics processing unit configured to perform dominant motion compensation comprising locally applied global motion compensation so that at least one other set of global motion trajectories are used at corners of at least one region on the frame that is less than the entire frame to form a displaced region; and use the pixel values of the displaced region to form a prediction region that corresponds to a region on the current frame;
the graphics processing unit configured to at least one of:
perform local global motion compensation on multiple regions of the frame by using a different set of global motion trajectories on each region;
wherein each region is a tile, and dividing the frame into the tiles, and wherein each tile has a set of global motion trajectories;
provide the option to perform local global motion compensation on a fraction of a tile in addition to entire tiles;
wherein each region is shaped and sized depending on an object associated with the region;
wherein the object is one of: a foreground, a background, and an object moving in the frame;
the graphics processing unit being configured to provide the option on the at least one region on a region-by-region basis to select a prediction formed by:
(1) a motion vector to form a prediction for the at least one region and using global motion compensation applied to the entire frame, or
(2) apply local global motion compensation with a set of global motion trajectories at the region and using displaced pixel values of the region to form a prediction;
the graphics processing unit configured to apply local global motion compensation with a set of global motion trajectories applied at a region of the reference frame that has an area less than the entire reference frame, and use motion vectors to form a prediction for the at least one region;
the graphics processing unit configured to provide the option to select a mode for a frame among:
(1) use the dominant motion compensated reference frame prediction,
(2) use blended prediction of multiple dominant motion compensated reference frames,
(3) use dominant motion compensated reference with differential translational motion vector for prediction, and
(4) use dominant motion compensated reference with differential translational motion vector for prediction, blended with another reference frame;
the graphics processing unit configured to at least one of (a) to (c):
(a) perform motion compensated morphed reference prediction using bilinear interpolation and motion compensation (MC) filter to form a morphed reference frame MRef, tPred h as the intermediate horizontal interpolation, and pred ji as the final motion compensated morphed reference prediction:
MRef
[
i
′
]
[
j
′
]
=
(
(
8
-
p
x
)
(
8
-
p
y
)
Ref
[
y
0
]
[
x
0
]
+
p
x
(
8
-
p
y
)
Ref
[
y
0
]
[
x
0
+
1
]
+
p
y
(
8
-
p
x
)
Ref
[
y
0
+
1
]
[
x
0
]
+
p
y
p
x
Ref
[
y
0
+
1
]
[
x
0
+
1
]
+
31
)
6
tPred
h
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
j
]
[
k
]
·
MRef
[
i
′
+
m
]
[
j
′
+
n
+
k
-
N
t
2
+
1
]
where
m
=
[
-
N
t
/
2
+
1
,
H
b
+
N
t
/
2
-
1
]
,
n
=
[
0
,
W
b
-
1
]
,
Pred
ji
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
i
]
[
k
]
·
tPred
h
[
m
+
k
-
N
t
2
+
1
]
[
n
]
where
m
=
[
0
,
H
b
-
1
]
,
n
=
[
0
,
W
b
-
1
]
,
and where:
(iMVx, iMVy) is the transmitted motion vector in Sub-Pel Unit (f s ) for a block at (j, i) of size (W b ×H b ); A, B, C, D, E, & F are affine parameters calculated from the three Motion trajectories transmitted; using separable motion compensation (MC) Filters with filter coefficients h[f s ][N i ] of norm T, fs is the Sub-Pel Factor (e.g. 2=Half Pel, 4=Quarter Pel, 8=Eighth Pel), where N i is the number MC Filter Taps, and
i′=i +( iMVy/f s )
j′=j +( iMVx/f s )
p i =iMVy & ( f s −1)
p j =iMVx & ( f s −1)
(j′, i′) is integer motion adjusted current pixel location in Morphed Reference Image; p j , p i are the ⅛ th pel phases in the Morphed Reference Image;
x =( A* 1+ B*i′+C<<r )>> r
y =( D*j′+E*i′+F<<s )>> s
where (x, y) is the reference pixel coordinate in ⅛ th Pel accuracy for location (j′, i′)
p y =y & 0×7
p x =x & 0×7
y 0 =>>3
x 0 =x>> 3
where (x 0 , y 0 ) is the integer pel location in Ref Image; p x , p y is the ⅛ th pel phase;
M Ref[ i′][j ′]=((8− p x )*(8− p y )*Ref[ y 0 ][x 0 ]+p x *(8− p y )*Ref[ y 0 ][x 0 +1]+ p y *(8− p x )*Ref[ y 0 +1][ x 0 ]+p y *p x *Ref[ y 0 +1][ x 0 +1]+31)>>6
t Pred h [m][n ]=SUM k ( h[p j ][k]*M Ref[ i′+m][j′+n+k ])/ T,
where m=[−N t /2−1, H b +N t /2], where n=[0, W b −1], where k=[−N t /2−1, N t /2],
Pred ji [m][n ]=SUM k [p j ][k]*t Pred h [m+k][n ])/ T,
where m=[0, H b −1], where n=[0, W b −1], where k=[−N t /2−1, +N t /2];
(b) perform morphed reference prediction using block motion compensation (MC) filtering to form a morphed reference frame Mref, and Predh as the intermediate horizontal interpolation:
tPred
h
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
x
]
[
k
]
·
Ref
[
y
0
+
m
]
[
x
0
+
n
+
k
-
N
t
2
+
1
]
m
=
[
-
N
t
/
2
+
1
,
H
s
+
N
t
/
2
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
MRef
[
i
+
m
]
[
j
+
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
y
]
[
k
]
·
tPred
h
[
m
+
k
-
N
t
2
+
1
]
[
n
]
where
m
=
[
0
,
H
s
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
and where A, B, C, D, E, & F are affine parameters calculated from the three Motion trajectories transmitted; using separable MC Filters with filter coefficients h[fs][N t ] of norm T; fs is the Sub-Pel Factor (e.g. 2=Half Pel, 4=Quarter Pel, 8=Eighth Pel), and where N t is the number MC Filter Taps
x =( A*j+B*i+C<<r )>> r
y =( D*j+E*i+F<<s )>> s
(j, i) is every (W s ×H s ) sub-block location in current image, x and y are reference pixel coordinates in ⅛th Pel accuracy;
p y =y & 0×7
p x =x & 0×7
y 0 =y>> 3
x 0 =x>> 3
(x 0 , y 0 ) is the integer pel location in the reference frame (Ref Image); p x , p y is the ⅛th pel phase;
t Pred h [m][n ]=SUM k ( h[p x ][k ]*Ref[ y 0 +m][x 0 +n+k ])/ T,
m=[−N t /2−1, H s +N t /2], n=[0, W s −1], k=[−N t /2−1, +N t /2]; and
M Ref[ i+m][j+n ]=SUM k ( h[p y ][k]*t Pred h [m+k][n ])/ T,
m=[0, H s −1], n=[0, W s −1], k=[−N t /2−1, +N t /2]; and
(c) perform motion compensated morphed reference prediction using single loop motion compensation (MC) filtering to form a morphed reference (Mref) and predictions tPred h as the intermediate horizontal interpolation, and Pred ji as the final motion compensated morphed reference prediction for block of size W b ×H b at (j, i):
tPred
h
[
m
]
[
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
x
]
[
k
]
·
Ref
[
y
0
+
m
]
[
x
0
+
n
+
k
-
N
t
2
+
1
]
for
:
m
=
[
-
N
t
/
2
+
1
,
H
s
+
N
t
/
2
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
Pred
ji
[
uH
s
+
m
]
[
vW
s
+
n
]
=
1
T
′
∑
k
=
0
N
t
-
1
h
[
p
y
]
[
k
]
·
tPred
h
[
m
+
k
-
N
t
2
+
1
]
[
n
]
for
:
m
=
[
0
,
H
s
-
1
]
,
n
=
[
0
,
W
s
-
1
]
,
u
=
[
0
,
H
b
/
H
s
-
1
]
,
v
=
[
0
,
W
b
/
W
s
-
1
]
,
and where:
(iMVx, iMVy) is the transmitted Motion Vector in Sub-Pel Units (fs) for a block at (j, i) of size (W b ×H b ); A, B, C, D, E, & F are affine parameters calculated from the three Motion trajectories transmitted; using separable MC Filters with filter coefficients h[fs][N t ] of norm T, fs is the Sub-Pel Factor (e.g. 2=Half Pel, 4=Quarter Pel, 8=Eighth Pel), and N t is the number MC Filter Taps;
i ′=( i+u*H s )* fs+iMVx
j ′=( j+v*W s )* fs+iMVy
where (j, i) is the current block pixel location, (u, v) is the index of every (W s ×H s ) sub-block within given current block of (W b ×H b ), and (W s ×H s ) sub-block; Below, i′, j′ is motion adjusted current pixel location in fs sub-pel accuracy,
x =(( A*j′+B*i ′+( C*fs )<< r )>>( r+ 3)
y =(( D*j′+E*i ′+( F*fs )<< s )>>( s+ 3)
where x & y are reference pixel coordinates in fs sub-pel accuracy
p y =y & ( f s −1)
p x =x & ( f s −1)
y 0 =y/fs
x 0 =x/fs
where y 0 , x 0 is the integer pel location in Ref Image, p x , p y is the ⅛th pel phase;
t Pred h [m][n ]=SUM k ( h[p x ][k ]*Ref[ y 0 +m][x 0 +n+k ])/ T,
m=[−N t /2−1, H s +N t /2],
n=[ 0, W s −1],
k=[−N t /2−1,+ N t /2],
Pred ji [u*H s +m][v*W s +n ]=SUM k ( h[p y ][k]*t Pred h [m+k][n ])/ T,
m=[ 0, H s −1],
n=[ 0, W s −1],
k=[−N t /2−1,+ N t /2],
V=[ 0, W b /W s −1],
u=[ 0, H b /H s −1].
72 . At least one computer readable memory comprising instructions, that when executed by a computing device, cause the computing device to:
obtain frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; divide the reference frame into a plurality of portions that are less than the area of the entire frame; perform dominant motion compensation comprising applying local global motion compensation on at least one of the portions by displacing the at least one portion of the decoded reference frame by using global motion trajectories at a boundary of the portion; and form a prediction portion that corresponds to a portion on the current frame and by using the pixel values of the displaced portion.
73 . The computer readable memory of claim 72 , wherein the instructions cause the computing device to:
perform local global motion compensation on a plurality of the portions by using a different set of global motion trajectories on each portion of the plurality of portions; wherein each portion is a tile, the instructions cause the computing device to divide the frame into the tiles, and wherein each tile has a set of global motion trajectories; the instructions cause the computing device to provide the option to perform local global motion compensation on a fraction of a tile in addition to entire tiles; wherein local global motion compensation trajectories are provided to half-tiles or quarter-tiles; the instructions cause the computing device to at least one of:
group a plurality of the tiles into a region, and apply the same global motion trajectories on the tiles within the same region, and different sets of global motion trajectories depending on the region; and
group a plurality of the portions into a region, and apply the same global motion trajectories on the portions within the same region, and different sets of global motion trajectories depending on the region;
wherein each portion is shaped and sized depending on an object associated with the portion; wherein the object is one of: a foreground, a background, and an object moving in the frame; wherein the portion is a rectangle placed about the object; the instructions cause the computing device to form a portion of the background of the reference frame, and a portion of the foreground of the reference frame each with a different set of local global motion trajectories for each portion.
74 . At least one computer readable memory comprising instructions, that when executed by a computing device, cause the computing device to:
obtain frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; form a warped global compensated reference frame by displacing at least one portion of the decoded reference frame by using global motion trajectories; determine a motion vector indicating the motion of the at least one portion and motion from a position based on the warped global compensated reference frame to a position at the current frame; and form a prediction portion based, at least in part, on the motion vectors and corresponding to a portion on the current frame.
75 . The computer readable memory of claim 74 , wherein the at least one portion is at least one of:
(1) a block of pixels used as a unit to divide the current frame and the reference frame into a plurality of the blocks; (2) at least one tile of pixels, each tile being at least 64×64 pixels, and used as a unit to divide the current frame and the reference frame into a plurality of the tiles;
the instructions causing the computing device to at least one of:
group tiles together based on common association with an object in the frame to form the at least one portion; and forming a single motion vector for each group of tiles,
group the tiles based on a merge map transmittable from an encoder to a decoder;
(3) a region of pixels shaped and sized depending on an object associated with the region, wherein a boundary of the region is at least one of:
a shape that resembles the shape of the object associated with the region, and
a rectangle placed around the object associated with the region;
wherein the region is associated with at least one of:
a background of the frame,
a foreground of the frame, and
a moving object in the frame;
the instructions causing the computing device to define the region based on a boundary map transmittable from an encoder to a decoder; wherein form a warped global compensated reference frame comprises using the global motion trajectories at the outer corners of the frame; wherein form a warped global compensated reference frame comprises using an affine or perspective global motion compensation method; wherein the at least one portion comprises a frame divided into a background and a foreground, and wherein determining motion vectors comprises providing the background and foreground each with one motion vector; the instructions causing the computing device to perform dominant motion compensation comprising locally applied global motion compensation so that at least one other set of global motion trajectories are used at corners of at least one region on the frame that is less than the entire frame to form a displaced region; and use the pixel values of the displaced region to form a prediction region that corresponds to a region on the current frame; the instructions causing the computing device to at least one of:
perform local global motion compensation on multiple regions of the frame by using a different set of global motion trajectories on each region;
wherein each region is a tile, and dividing the frame into the tiles, and wherein each tile has a set of global motion trajectories;
provide the option to perform local global motion compensation on a fraction of a tile in addition to entire tiles;
wherein each region is shaped and sized depending on an object associated with the region; wherein the object is one of: a foreground, a background, and an object moving in the frame; the instructions causing the computing device to provide the option on the at least one region on a region-by-region basis to select a prediction formed by:
(1) a motion vector to form a prediction for the at least one region and use global motion compensation applied to the entire frame, or
(2) apply local global motion compensation with a set of global motion trajectories at the region and using displaced pixel values of the region to form a prediction;
the instructions causing the computing device to apply local global motion compensation with a set of global motion trajectories applied at a region of the reference frame that has an area less than the entire reference frame, and use motion vectors to form a prediction for the at least one region; and the instructions causing the computing device to provide the option to select a mode for a frame among:
(1) use the dominant motion compensated reference frame prediction,
(2) use blended prediction of multiple dominant motion compensated reference frames,
(3) use dominant motion compensated reference with differential translational motion vector for prediction, and
(4) use dominant motion compensated reference with differential translational motion vector for prediction, blended with another reference frame.Join the waitlist — get patent alerts
Track US2017013279A1 — get alerts on status changes and closely related new filings.
We store only your email — no account needed. See our privacy policy.