US11397890B2ActiveUtilityA1
Cross-media retrieval method based on deep semantic space
Assignee: UNIV PEKING SHENZHEN GRADUATE SCHOOLPriority: Apr 10, 2017Filed: Aug 16, 2017Granted: Jul 26, 2022
Est. expiryApr 10, 2037(~10.8 yrs left)· nominal 20-yr term from priority
Inventors:Wenmin WangMengdi FanPeilei DongRonggang WangGe LiShengfu DongZhenyu WangYing LiHui ZhaoWen Gao
G06N 3/048G06N 3/045G06N 3/044G06N 5/022G06N 3/09G06N 3/0464G06N 3/0442G10L 15/16G06N 3/08G06F 16/583G06F 16/483G06N 3/0445G06N 3/0454
38
PatentIndex Score
0
Cited by
5
References
7
Claims
Abstract
The present application discloses a cross-media retrieval method based on deep semantic space, which includes a feature generation stage and a semantic space learning stage. In the feature generation stage, a CNN visual feature vector and an LSTM language description vector of an image are generated by simulating a perception process of a person for the image; and topic information about a text is explored by using an LDA topic model, thus extracting an LDA text topic vector. In the semantic space learning phase, a training set image is trained to obtain a four-layer Multi-Sensory Fusion Deep Neural Network, and a training set text is trained to obtain a three-layer text semantic network, respectively. Finally, a test image and a text are respectively mapped to an isomorphic semantic space by using two networks, so as to realize cross-media retrieval. The disclosed method can significantly improve the performance of cross-media retrieval.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1. A cross-media retrieval method based on deep semantic space, comprising:
mining semantic information in cross-media retrieval by simulating a perception process of a person for the image, to accomplish cross-media retrieval, which comprises a feature generation process and a semantic space learning process,
the cross-media retrieval method further comprising:
Step 1) obtaining training data, test data and data categories;
Step 2) in the feature generation process, extracting features for images and text respectively, comprising:
Step 21) generating a CNN visual feature vector and an LSTM language description vector of an image for training and test images by using the Convolutional Neural Network-Long Short Term Memory (CNN-LSTM);
for the N training images, obtaining the features of each image (CNN visual feature vector, LSTM language description vector, real tag value ground-truth label), which is expressed as D=(v (n) ,d (n) ,l (n) ) n=1 N ,. l represents the l-th layer of the Convolutional Neural Network, and l≥2; and
Step 22) using the document topic to generate the model LDA, and extracting the “LDA text topic vector” of the training and test text; for N training texts, the “LDA text topic vector” extracted for each sample is represented as t;
Step 3) in the semantic space learning process comprising a semantic space learning process of images and a semantic space learning process of texts, respectively mapping images and texts into a common semantic space;
Step 31) in the semantic space learning process of images: constructing a four-layer Multi-Sensory Fusion Deep Neural Network MSF-DNN for semantic space learning, and obtaining a parameter space Ω=(W A (l) ,b A (l) ) where W A (l) represents the weight matrix, b A (l) represents the offset, and l represents the number of layers; and
Step 32) in the semantic space learning process of texts, constructing a three-layer text semantic network TextNet for semantic space learning, and a parameter space Ω′=(W t (l′) ,b t (l′) ) is obtained; W t (l′) represents the weight matrix; b t (l′) represents the offset; and l′ represents the number of layers of TextNet, which map the image and text to an isomorphic semantic space through MSF-DNN and TextNet; and
Step 4) calculating the similarity between any image and text using a similarity measurement method, and accomplishing the cross-media retrieval of the Image Retrieval in Text (Img2Text) and Text Retrieval in Image (Text2Img).
2. A cross-media retrieval method according to claim 1 , wherein in Step 1), training data, test data, and obtaining data categories through the data sets Wikipedia, Pascal Voc, and Pascal Sentence.
3. A cross-media retrieval method according to claim 1 , wherein in Step 21), generating a CNN visual feature vector and a LSTM language description vector of an image for training and test images by using the Convolutional Neural Network-Long Short Term Memory (CNN-LSTM) architecture, wherein the CNN network is fine-adjusted by using the training image of the existing data set, and then the output of the last 1,024-dimensional full connection layer is extracted for the training image and the test image as “CNN visual feature vector”,
wherein the extraction of LSTM language description vector includes: in Long Short Term Memory (LSTM), when t is equal to the last time N, extracting the tuple (C N , h N ) as the “LSTM language description vector” of the training image and the test image.
4. A cross-media retrieval method according to claim 1 , wherein in Step 22), optimal numbers of topics selected for the three data sets of Wikipedia, Pascal Voc, and Pascal Sentence are 200, 100, and 200, respectively.
5. A cross-media retrieval method according to claim 1 , wherein in Step 31), the semantic space learning process for the image comprises:
Step 311) for N training images, generating features after Step 21, and getting the features of each picture, expressed as D=(v (n) ,d (n) ,l (n) ) n=1 N , where l represents the l-th layer (l≥2) of the neural network, x j denotes the input vector of the l−1-th layer, wherein the value z i (l) before the i-th activation of the l-th layer is expressed as the Formula 1:
z i (l) =Σ j=1 m W ij (l-1) x j +b i (l-1) (1)
where m is the number of units in the l−1-th layer; W ij (l-1) represents the weight between the j-th unit of the l−1-th layer and the i-th unit of the l-th layer; and b i (l-1) represents the weight associated with the i-th unit of the l-th layer;
Step 312) calculating the activation value f I (l) (z) for each z by Formula 2, where the activation function of all hidden layers uses the Sigmoid function, the last output layer uses the Softmax function for activation:
f
I
(
l
)
(
z
)
=
{
1
/
(
1
+
e
-
z
)
l
=
2
,
3
e
(
z
-
ɛ
)
/
∑
k
=
1
K
e
(
z
k
-
ɛ
)
l
=
4
(
2
)
where l represents the number of layers, K is the number of labels, and ε=max(z k );
Step 313) defining the MSF-DNN network as Formula 3-6:
h v (2) =f I (2) ( W v (1) ·v+b v (1) ) (3)
h d (2) =f I (2) ( W d (1) ·d+b d (1) ) (4)
h c (3) =f I (3) ( W c (2) ·[h v (2) , h d (2) ]+b c (2) ) (5)
o I =f I (4) ( W c (3) ·h c (3) +b c (3) ) (6)
where h A (l) represents a hidden layer with a depth of l, o I represents the last layer of the output layer; W A (l) represents a weight matrix; b A (l) represents an offset; when l=1, A=v or d, otherwise A=c; and c is the output after the fusion of the two values; and
Step 314) minimizing an overall error C of the training sample using an objective function to learn to obtain a parameter space Ω=(W A (l) ,b A (l) , expressed as Formula 7:
C
=
arg
min
Ω
1
2
N
∑
n
=
1
N
o
I
(
n
)
-
l
(
n
)
2
+
λ
1
2
∑
l
=
1
3
W
A
(
l
)
F
2
(
7
)
where λ I is a parameter of the second weight attenuation term.
6. A cross-media retrieval method according to claim 1 , wherein in Step 32), the semantic space learning process of texts comprises:
Step 321) for N training text samples, wherein the “LDA text subject vector” of each sample is represented as t, the full connection layer of the second layer uses the Sigmoid activation function and then uses the output as the input to the last layer of the Softmax classifier, expressing a definition of the TextNet network using Formula 8 and Formula 9:
h t (2) =f T (2) ( W t (1) ·t+b t (1) ) (8)
o T =f T (3) ( W t (2) ·h t (2) +b t (2) ) (9)
where h t (2) represents the second layer of hidden layer, o T represents the last layer of the) output layer; W t (l′) represents the weight matrix; b t (l′) represents the offset, and l′ represents the number of layers of TextNe; and
Step 322) minimizing an overall error C of the training sample using an objective function to learn to obtain a parameter space Ω=(W t (l′) ,b t (l′) ), expressed as Formula 10:
C
′
=
arg
min
Ω
′
1
2
N
∑
n
=
1
N
o
T
(
n
)
-
l
(
n
)
2
+
λ
T
2
∑
l
=
1
2
W
t
(
l
′
)
F
2
(
10
)
in which λ T is the parameter of the second weight attenuation term.
7. A cross-media retrieval method according to claim 1 , wherein in Step 4), the similarity measurement method uses a cosine distance to represent the similarity between any image and any text feature, and for an image vector S I ∈R K , and the text vector S T ∈R K , the cosine distance d(S I , S T ) is calculated by Formula 11:
d
(
S
I
,
S
T
)
=
∑
k
=
1
K
S
I
(
k
)
×
S
T
(
k
)
∑
k
=
1
K
S
I
(
k
)
2
×
∑
k
=
1
K
S
T
(
k
)
2
(
11
)
where K is the dimension of the feature, the calculatedd (S I , S T ) is taken as the similarity between the image and the text features. Sort by similarity from high to low, the top k samples with the highest similarity is taken as the retrieval result, to accomplish cross-media retrieval.Cited by (0)
No later patents cite this yet.
References (0)
No backward citations on record.