US2013282687A1PendingUtilityA1
System and method for multimedia information retrieval
Est. expiryDec 15, 2030(~4.4 yrs left)· nominal 20-yr term from priority
G06F 18/256G06F 17/18G06F 16/43G06F 17/30023G06F 17/30864G06F 16/432G06F 16/951
51
PatentIndex Score
0
Cited by
0
References
0
Claims
Abstract
A method for information retrieval includes querying a multimedia collection with a first component of a multimedia query to generate first comparison measures between the first component of the query and respective objects in the collection for a first media type. The collection is queried with a second component of the multimedia query to generate second comparison measures between the second component of the query and respective objects in the collection for a second media type. An aggregated score for each of a set of objects in the collection is computed by applying an aggregating function in which a first confidence weighting is applied to the first comparison measure and a second confidence weighting is applied to the second comparison measure. The first confidence weighting is independent of the second comparison measure. The second confidence weighting is dependent on the first comparison measure.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1 . A method for multimedia information retrieval comprising:
querying a multimedia collection with a first component of a multimedia query to generate first comparison measures between the first query component and respective objects in the collection for a first media type; querying at least a part of the multimedia collection with a second component of the multimedia query to generate second comparison measures between the second query component and respective objects in the collection for a second media type; computing an aggregated score for each of a set of objects in the collection based on the first comparison measure and the second comparison measure for the object, the computing comprising applying an aggregating function to aggregate the first and second comparison measures, wherein in the aggregating function, a first confidence weighting is applied to the first comparison measure and a second confidence weighting is applied to the second comparison measure, the first confidence weighting is independent of the second comparison measure and the second confidence weighting is dependent on the first comparison measure; and outputting information based on the aggregated scores.
2 . The method of claim 1 , wherein the computing of the aggregated score is performed with a computer processor.
3 . The method of claim 1 , wherein the first media type comprises text and the first component of the multimedia query is a text-based query component.
4 . The method of claim 3 , wherein the text-based query component comprises a representation of a text document based on word frequencies.
5 . The method of claim 1 , wherein the second media type comprises images and the second component of the multimedia query is an image-based query component.
6 . The method of claim 5 , wherein the image-based query component comprises a representation of an image which is based on extracted features.
7 . The method of claim 1 , wherein the first confidence weighting is dependent of the first comparison measure and the second confidence weighting is dependent on the first and second comparison measures.
8 . The method of claim 1 , wherein:
the first comparison measure comprises at least one of a first similarity score and a first ranking based on the first similarity score with respect to first similarity scores of other objects in the collection; and the second comparison measure comprises at least one of a second similarity score and a second ranking based on the second similarity score with respect to second similarity scores of other objects in the collection.
9 . The method of claim 8 , wherein in the computing of the aggregated score, the first confidence weighting applied to the first comparison measure is dependent on at least one of the first similarity score and the first ranking.
10 . The method of claim 8 , wherein in the computing of the aggregated score, the second confidence weighting applied to the second comparison measure is dependent on at least one of the first similarity score and the first ranking.
11 . The method of claim 1 , wherein the aggregating function is a linear combination of a function of the confidence weighted first comparison measure and a function of the confidence weighted second comparison measure.
12 . The method of claim 1 , wherein the aggregated score is computed using an aggregating function of general format:
s cw ( q,o )= A ( N ( s a ( q,o )ƒ), N ( s b ( q,o ) g )) (3)
where s cw is the aggregated score; A represents the aggregating function; a represents the first media type; b represents the second media type; s a (q,o) and s b (q,o) are similarity scores between the query q and the object o for the first and second media types respectively; r a (o,q) and r b (o,q) are rankings of the object o given by the respective similarity scores s a (q,o) and s b (q,o), with respect to other objects in the collection; ƒ is a function of at least one of r a (o,q) and s a (q,o) and optionally also of θ a ; g is a function of at least one of r a (o,q) and s a (q,o) and at least one of s b (q,o) and r b (o,q) and optionally also of θ b ; θ a is a set of one or more parameters; θ b is a set of one or more parameters;
and
N represents an optional normalizing operator.
13 . The method of claim 12 , wherein:
ƒ is a function of r a (o,q); and g is a function of r a (o,q).
14 . The method of claim 13 , wherein at least one of:
f
(
r
t
(
o
,
q
)
)
=
g
(
r
t
(
o
,
q
)
)
=
{
1
if
r
t
(
o
,
q
)
≤
K
0
otherwise
;
f
(
r
t
(
o
,
q
)
)
=
g
(
r
t
(
o
,
q
)
)
=
{
log
(
K
-
r
t
(
o
,
q
)
+
1
)
log
(
K
)
if
r
t
(
o
,
q
)
<
K
0
otherwise
;
f
(
r
t
(
o
,
q
)
)
=
g
(
r
t
(
o
,
q
)
)
=
{
x
(
K
+
1
)
(
K
+
r
t
(
o
,
q
)
)
-
y
if
r
t
(
o
,
q
)
<
K
0
otherwise
;
and
f
(
r
t
(
o
,
q
)
)
=
g
(
r
t
(
o
,
q
)
)
=
{
s
t
(
o
,
q
)
δ
if
r
t
(
o
,
q
)
<
K
0
otherwise
is applied in the aggregating function, where K represents a number of retrieved objects, and x, y and δ are each a parameter whose value is preselected.
15 . The method of claim 14 , wherein x is at least 1, y is at least 0, and δ is greater than 0.
16 . The method of claim 12 , wherein the aggregating function has the general format:
s
cw
(
q
,
o
)
=
α
a
N
(
s
a
(
q
,
o
)
f
(
s
a
(
q
,
o
)
,
r
a
(
o
,
q
)
,
θ
a
)
)
+
α
b
N
(
s
b
(
q
,
o
)
g
(
s
a
(
q
,
o
)
,
r
a
(
o
,
q
)
,
s
b
(
q
,
o
)
,
r
b
(
o
,
q
)
,
θ
b
)
(
4
)
where 0<α a <1 and 0<α b <1.
17 . The method of claim 16 , wherein α b =1−α a .
18 . The method of claim 1 , wherein the aggregating function comprises an average mean operator.
19 . The method of claim 1 , wherein the outputting information comprises outputting objects from the collection based on their aggregated scores.
20 . A system for performing the method of claim 1 , comprising non-transitory memory which stores instructions for performing the method and a processor in communication with the memory for executing the instructions.
21 . A computer program product comprising a non-transitory recording medium encoding instructions, which when executed on a computer causes the computer to perform the method of claim 1 .
22 . A system for multimedia information retrieval comprising:
a search engine for querying an associated multimedia collection with a first component of a multimedia query and for querying at least a part of the queried multimedia collection with a second component of the multimedia query; a first comparison component for generating a first comparison measure between the first query component and a respective object in the collection for a first media type; a second comparison component for generating a second comparison measure between the second query component and a respective object in the collection for the second media type; a multimedia scoring component for generating aggregated scores for each of a set of objects in the collection based on the first comparison measure and the second comparison measure for the respective object, the multimedia scoring component applying an aggregating function to the first and second comparison measure in which a first confidence weighting is applied to the first comparison measure which is independent of the second comparison measure and a second confidence weighting is applied to the second comparison measure which is dependent on the first comparison measure; and a processor for implementing the search engine, first and second comparison components, and multimedia scoring component.
23 . The system of claim 22 , further comprising the collection of multimedia objects.
24 . The system of claim 22 , wherein the system is configured for outputting retrieved objects from the collection based on their aggregated scores.
25 . The system of claim 22 , wherein the system is configured for receiving a multimedia query from a user which includes a textual component and an image component.
26 . An information retrieval method comprising:
retrieving a set of objects from a collection based on computed text-based scores for the objects; computing image-based scores for the retrieved objects; with a processor, computing an aggregated score for objects in the retrieved set of objects based on the respective text-based and image-based scores in which a first confidence weighting is applied to the text-based score which is independent of the image-based score and a second confidence weighting is applied to the image-based score which is dependent on the text-based score; and outputting information based on the aggregated scores.Cited by (0)
No later patents cite this yet.
References (0)
No backward citations on record.