US10341634B2ActiveUtilityA1
Method and apparatus for acquiring image disparity
Est. expiryJan 29, 2036(~9.6 yrs left)· nominal 20-yr term from priority
H04N 13/239H04N 2013/0081H04N 13/122H04N 13/128
45
PatentIndex Score
0
Cited by
14
References
19
Claims
Abstract
A method and apparatus for acquiring an image disparity are provided. The method may include acquiring, from dynamic vision sensors, a first image having a first view of an object and a second image having a second view of the object; calculating a cost within a preset disparity range of an event of first image and a corresponding event of the second image; calculating an intermediate disparity of the event of the first image and an intermediate disparity of the event of the second image based on the cost; determining whether the event of the first image is a matched event based on the intermediate disparity of the event of the first image and the intermediate disparity of the event of the second image; and predicting optimal disparities of all events of the first image based on an intermediate disparity of the matched event of the first imaged.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1. A method of acquiring an image disparity by one or more hardware processors, the method comprising:
acquiring, from dynamic vision sensors, a first image having a first view of an object and a second image having a second view of the object;
calculating a cost within a preset disparity range of an event of first image and a corresponding event of the second image, wherein the event of the first image and the corresponding event of the second image are generated when an intensity of lighting is greater than a preset threshold;
calculating an intermediate disparity of the event of the first image and an intermediate disparity of the event of the second image based on the cost;
determining whether the event of the first image is a matched event based on the intermediate disparity of the event of the first image and the intermediate disparity of the event of the second image; and
in response to the event of the first image being determined as the matched event, predicting optimal disparities of all events of the first image based on the intermediate disparity of the event of the first image.
2. The method of claim 1 , further comprising:
removing noise from the first image to acquire a noise-removed first image.
3. The method of claim 2 , wherein the removing comprises:
acquiring {δ 1 u 1 v 1 H , δ 2 H 2 v 2 H , . . . , δ l u l v l H , . . . , δ r u r v r H } by performing feature decomposition with respect to the first image, δ t denoting an i-th eigenvalue, δ 1 , δ 2 , . . . , δ r denoting eigenvalues arranged in descending order, u i and v i H denoting eigenvectors orthogonal to each other, and r denoting a total number of the eigenvalues;
acquiring first to k-th eigenvalues δ 1 , δ 2 , . . . , δ k which are greater than or equal to a preset value, among δ 1 , δ 2 , . . . , δ r , k being less than r; and
acquiring the noise-removed image based on the following equation:
I
0
=
∑
i
=
1
k
δ
i
u
i
v
i
H
wherein I 0 denotes the noise-removed first image.
4. The method of claim 1 , wherein the calculating the cost comprises:
calculating a feature of a pixel of the first image and the second image; and
calculating the cost based on the feature of the pixel within a local range of a center point of the first and second images.
5. The method of claim 4 , wherein the calculating the feature of the pixel comprises:
calculating the feature of the pixel based on the following equation:
F
n
(
x
,
y
)
=
min
i
(
abs
(
x
-
i
)
❘
rotE
n
(
i
,
y
)
≠
0
)
wherein F n (x,y) denotes the feature of the pixel in an n-th direction, x and y denote an x-axial coordinate and a y-axial coordinate of the pixel (x,y), respectively, n denotes a value ranging from 1 to N, N denotes a total number of directions, i≠x is satisfied, E n (i,y) denotes a polarity of a pixel (i,y) near the pixel in the n-th direction, i denotes an x-axial coordinate of the pixel (i,y), rot denotes a function to rotate, abs denotes a function to obtain an absolute value, and min denotes a function to obtain a minimum value.
6. The method of claim 4 , wherein the calculating the cost based on the feature of the pixel within the local range of the center point comprises:
calculating a final cost matrix corresponding to the cost based on the following equation:
{
C
(
x
,
y
,
d
)
=
α
·
∑
n
=
1
N
CF
n
(
x
,
y
,
d
)
+
(
1
-
α
)
·
CP
(
x
,
y
,
d
)
CF
n
(
x
,
y
,
d
)
=
∑
(
x
′
,
y
′
)
∈
W
(
x
,
y
)
[
F
n
(
x
′
+
d
,
y
′
)
-
F
n
(
x
′
,
y
′
)
]
CP
(
x
,
y
,
d
)
=
[
E
(
x
+
d
,
y
)
-
E
(
x
,
y
)
]
2
wherein d denotes a disparity value within a preset disparity range, C(x,y,d) denotes the final cost matrix, CF n (x,y,d) denotes a feature matching cost, CP(x,y,d) denotes a polarity matching cost, α denotes a linear combination weight, x and y denote an x-axial coordinate and a y-axial coordinate of the pixel (x,y), respectively, W(x,y) denotes the local range of the center point, (x′,y′) denotes an arbitrary pixel within W(x,y), N denotes a total number of directions, n denotes a value ranging from 1 to N, F n (x′,y′) denotes a feature of the arbitrary pixel (x′,y′) in an n-th direction, F n (x′+d,y′) denotes a feature of a pixel (x′+d,y′) in the n-th direction, E(x,y) denotes a polarity of the pixel (x,y), and E(x+d,y) denotes a polarity of a pixel (x+d,y).
7. The method of claim 6 , wherein the calculating the intermediate disparity of the event of the first image comprises:
calculating the intermediate disparity of the event of the first image based on the following equation:
D
(
x
,
y
)
=
arg
min
d
C
(
x
,
y
,
d
)
wherein D(x,y) denotes the intermediate disparity of the event of the first image, and arg min C(x,y,d) denotes d that minimizes C(x,y,d).
8. The method of claim 1 , wherein the determining comprises:
verifying whether the intermediate disparity of the event of the first image is equal to the intermediate disparity of the corresponding event of the second image;
determining the event of the first image to be the matched event when the intermediate disparity of the event of the first image is equal to the intermediate disparity of the corresponding event of the second image; and
determining the event of the first image to be an unmatched event when the intermediate disparity of the event of the first image is different from the intermediate disparity of the corresponding event of the second image.
9. The method of claim 1 , wherein the predicting comprises:
calculating a set of the optimal disparities of all the events of the first image based on the following equation:
D
^
=
arg
min
D
^
(
λ
·
∑
i
=
1
M
(
d
^
i
-
d
i
)
+
(
1
-
λ
)
∑
j
=
1
S
(
d
^
j
-
∑
m
=
1
K
w
jm
·
d
m
)
)
wherein {circumflex over (D)} denotes the set of optimal disparities of all the events of the first λ image, denotes a linear combination weight, M denotes a total number of matched events of the first image, d i denotes an intermediate disparity of a matched event i of the first image calculated based on the cost within the preset disparity range, {circumflex over (d)} i denotes an optimal disparity of the matched event i of the first image, S denotes a total number of events of the first image, {circumflex over (d)} j denotes an optimal disparity of an event j of the first image, K denotes a total number of matched events selected near the event j, d m denotes an intermediate disparity of a matched event m calculated based on the cost within the disparity range, among the selected matched events, w jm denotes a feature similarity between the event j and the matched event m, and
arg
min
D
^
(
λ
·
∑
i
=
1
M
(
d
^
i
-
d
i
)
+
(
1
-
λ
)
∑
j
=
1
S
(
d
^
j
-
∑
m
=
1
K
w
jm
·
d
m
)
)
denotes {circumflex over (D)} that minimizes
λ
·
∑
i
=
1
M
(
d
^
i
-
d
i
)
+
(
1
-
λ
)
∑
j
=
1
S
(
d
^
j
-
∑
m
=
1
K
w
jm
·
d
m
)
.
10. The method of claim 1 , wherein the second image corresponds to a right image of the object when the first image corresponds to a left image of the object, and the second image corresponds to the left image when the first image corresponds to the right image.
11. An apparatus for acquiring an image disparity, the apparatus comprising one or more hardware processors comprising:
a cost calculator configured to calculate a cost within a preset disparity range of an event of a first image and a corresponding event of a second image, wherein the event of the first image and the corresponding event of the second image are generated when an intensity of lighting is greater than a preset threshold;
a disparity calculator configured to calculate an intermediate disparity of the event of the first image and an intermediate disparity of the event of the second image based on the cost;
a determiner configured to determine whether the event of the first image is a matched event based on the intermediate disparity of the event of the first image and the intermediate disparity of the event of the second image; and
a disparity predictor configured to predict optimal disparities of all events of the first image based on the intermediate disparity of the event of the first image, in response to the event of the first image being determined as the matched event.
12. The apparatus of claim 11 , wherein the cost calculator comprises:
a noise remover configured to remove noise from the first image to acquire a noise-removed first image.
13. The apparatus of claim 12 , wherein the noise remover comprises:
a decomposer configured to acquire {δ 1 u 1 v 1 H , δ 2 H 2 v 2 H , . . . , δ l u l v l H , . . . , δ r u r v r H } by performing feature decomposition with respect to the first image, δ t denoting an i-th eigenvalue, δ 1 , δ 2 , . . . , δ r denoting eigenvalues arranged in descending order, u i and V i H denoting eigenvectors orthogonal to each other, and r denoting a total number of the eigenvalues;
an acquirer configured to acquire first to k-th eigenvalues δ 1 , δ 2 , . . . , δ k which are greater than or equal to a preset value, among δ 1 , δ 2 , . . . , δ r , k being less than r; and
an image calculator configured to acquire the noise-removed image based on the following equation:
I
0
=
∑
i
=
1
k
δ
i
u
i
v
i
H
wherein I 0 denotes the noise-removed first image.
14. The apparatus of claim 11 , wherein the cost calculator comprises:
a feature calculator configured to calculate a feature of a pixel of first image and the second image; and
a cost determiner configured to calculate the cost based on the feature of the pixel within a local range of a center point of the first and second images.
15. The apparatus of claim 14 , wherein the feature calculator is configured to calculate the feature of the pixel based on the following equation:
F
n
(
x
,
y
)
=
min
i
(
abs
(
x
-
i
)
❘
rotE
n
(
i
,
y
)
≠
0
)
wherein F n (x,y) denotes the feature of the pixel in an n-th direction, x and y denote an x-axial coordinate and a y-axial coordinate of the pixel (x,y), respectively, n denotes a value ranging from 1 to N, N denotes a total number of directions, i≠x is satisfied, E n (i,y) denotes a polarity of a pixel (i,y) near the pixel in the n-th direction, i denotes an x-axial coordinate of the pixel (i,y), rot denotes a function to rotate, abs denotes a function to obtain an absolute value, and min denotes a function to obtain a minimum value.
16. The apparatus of claim 14 , wherein the cost determiner is configured to calculate a final cost matrix corresponding to the cost based on the following equation:
{
C
(
x
,
y
,
d
)
=
α
·
∑
n
=
1
N
CF
n
(
x
,
y
,
d
)
+
(
1
-
α
)
·
CP
(
x
,
y
,
d
)
CF
n
(
x
,
y
,
d
)
=
∑
(
x
′
,
y
′
)
∈
W
(
x
,
y
)
[
F
n
(
x
′
+
d
,
y
′
)
-
F
n
(
x
′
,
y
′
)
]
CP
(
x
,
y
,
d
)
=
[
E
(
x
+
d
,
y
)
-
E
(
x
,
y
)
]
2
wherein d denotes a disparity value within a preset disparity range, C(x,y,d) denotes the final cost matrix, CF n (x,y,d) denotes a feature matching cost, CP(x,y,d) denotes a polarity matching cost, α denotes a linear combination weight, x and y denote an x-axial coordinate and a y-axial coordinate of the pixel (x,y), respectively, W(x,y) denotes the local range of the center point, (x′,y′) denotes an arbitrary pixel within, W(x,y) denotes a total number of directions, n denotes a value ranging from 1 to N, F n (x′,y′) denotes a feature of the arbitrary pixel (x′,y′) in an n-th direction, F n (x′+d,y′) denotes a feature of a pixel (x′+d,y′) in the n-th direction, E(x,y) denotes a polarity of the pixel (x,y), and E(x+d,y) denotes a polarity of a pixel (x+d,y).
17. The apparatus of claim 16 , wherein the disparity calculator is configured to calculate the intermediate disparity of the event of the first image based on the following equation:
D
(
x
,
y
)
=
arg
min
d
C
(
x
,
y
,
d
)
wherein D(x,y) denotes the intermediate disparity of the event of the first image, and arg min C(x,y,d) denotes d that minimizes C(x,y,d).
18. The apparatus of claim 11 , wherein the determiner comprises:
a verifier configured to verify whether the intermediate disparity of the event of the first image is equal to the intermediate disparity of the corresponding event of the second image; and
an event determiner configured to determine the event of the first image to be the matched event when the verifier verifies that the intermediate disparity of the event of the first image is equal to the intermediate disparity of the corresponding event of the second image, and determine the event of the first image to be an unmatched event when the verifier verifies that the intermediate disparity of the event of the first image is different from the intermediate disparity of the corresponding event of the second image.
19. The apparatus of claim 11 , wherein the disparity predictor is configured to calculate a set of the optimal disparities of all the events of the first image based on the following equation:
D
^
=
arg
min
D
^
(
λ
·
∑
i
=
1
M
(
d
^
i
-
d
i
)
+
(
1
-
λ
)
∑
j
=
1
S
(
d
^
j
-
∑
m
=
1
K
w
jm
·
d
m
)
)
wherein {circumflex over (D)} denotes the set of optimal disparities of all the events of the first image, λ denotes a linear combination weight, M denotes a total number of matched events of the first image, d i denotes an intermediate disparity of a matched event i of the first image calculated based on the cost within the preset disparity range, {circumflex over (d)} i denotes an optimal disparity of the matched event i of the first image, S denotes a total number of events of the first image, {circumflex over (d)} j denotes an optimal disparity of an event j of the first image, K denotes a total number of matched events selected near the event j, d m denotes an intermediate disparity of a matched event m calculated based on the cost within the disparity range, among the selected matched events, w jm denotes a feature similarity between the event j and the matched event m, and
arg
min
D
^
(
λ
·
∑
i
=
1
M
(
d
^
i
-
d
i
)
+
(
1
-
λ
)
∑
j
=
1
S
(
d
^
j
-
∑
m
=
1
K
w
jm
·
d
m
)
)
denotes {circumflex over (D)} that minimizes
λ
·
∑
i
=
1
M
(
d
^
i
-
d
i
)
+
(
1
-
λ
)
∑
j
=
1
S
(
d
^
j
-
∑
m
=
1
K
w
jm
·
d
m
)
.Cited by (0)
No later patents cite this yet.
References (0)
No backward citations on record.