Information processing device, information processing method and program

Abstract

An information processing device, method and a program adds an annotation to content and provides an application which utilizes the annotation. A learning module extracts an image feature amount of each frame of an image of learning content and extracts word frequency information regarding a frequency of appearance of each word in a description text, and learns an annotation models which is a multi-stream Hidden Markov Model (HMM) by using a multi-stream including the image feature amount and the text feature amount. A browsing controller extracts a scene which is a group of one or more temporally continuous frames from target content by using the annotation model and displays representative images of the scenes so as to be arranged in chronological order.

The invention claimed is: 
     
       1. An information processing device, comprising:
 a learning module configured to 
 extract an image feature amount of each frame of an image of learning content and extracting word frequency information regarding a frequency of appearance of each word in a description text describing a content of the image of the learning content as a text feature amount of the description text,
 wherein the learning content includes a text of a caption, and 
 wherein the description text is the text of the caption included in the learning content, and 
 
 learn an annotation model, which is a multi-stream HMM (Hidden Markov Model), by using an annotation sequence for annotation, which is a multi-stream including the image feature amount and the text feature amount; and 
 a browsing controller configured to extract a scene, which is a group of one or more temporally continuous frames, from target content from which the scene is to be extracted by using the annotation model, and
 display representative images of scenes so as to be arranged in chronological order. 
 
 
     
     
       2. The information processing device according to  claim 1 , wherein
 the learning means 
 extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals, and 
 extracts multinomial distribution, which represents a frequency of appearance of each word in the document, as the text feature amount. 
 
     
     
       3. The information processing device according to  claim 1 , wherein
 the learning means 
 extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals, and 
 extracts multinomial distribution, which represents a frequency of appearance of each word in the document, as the text feature amount, and 
 the browsing controlling means 
 extracts the image feature amount of each frame of the image of the target content and composes the annotation sequence by using the image feature amount, 
 obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model, 
 selects a word with high frequency in the multinomial distribution observed in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence as an annotation to be added to the frame of interest, 
 extracts a group of one or more temporally continuous frames to which the same annotation is added as the scene from the target content, and 
 displays the representative images of the scenes so as to be arranged in chronological order. 
 
     
     
       4. The information processing device according to  claim 3 , wherein
 the target content is content of a broadcast program, and 
 the browsing controlling means displays the representative images of the scenes of the broadcast program so as to be arranged in chronological order in a program listing of the broadcast program on an EPG (electronic program guide). 
 
     
     
       5. The information processing device according to  claim 3 , wherein
 the browsing controlling means also displays the annotation added to the frame, which composes the scene, together with a representative image of the scene. 
 
     
     
       6. The information processing device according to  claim 1 , wherein
 the learning means 
 extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals, and 
 extracts multinomial distribution, which represents a frequency of appearance of each word in the document, as the text feature amount, and 
 the browsing controlling means 
 extracts the image feature amount of each frame of the image of the target content and composes the annotation sequence by using the image feature amount, 
 obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model, 
 selects, when a frequency of a predetermined keyword is high in the multinomial distribution observed in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence, the frame of interest as a keyword frame, which is the frame of which content coincides with a predetermined keyword, 
 extracts a group of one or more temporally continuous frames as the scene from the keyword frame, and 
 displays the representative images of the scenes so as to be arranged in chronological order. 
 
     
     
       7. The information processing device according to  claim 6 , wherein
 the target content is content of a broadcast program, and 
 the browsing controlling means displays the representative images of the scenes of the broadcast program so as to be arranged in chronological order in a program listing of the broadcast program on an EPG (electronic program guide). 
 
     
     
       8. The information processing device according to  claim 1 , wherein
 the learning means 
 performs dimension reduction to reduce a dimension of the image feature amount and the text feature amount, and 
 learns the annotation model by using the multi-stream including the image feature amount and the text feature amount after the dimension reduction as the annotation sequence. 
 
     
     
       9. The information processing device according to  claim 8 , wherein the learning means
 obtains basis space data of a basis space for image of which dimension is lower than the dimension of the image feature amount for mapping the image feature amount by using the image feature amount, 
 performs the dimension reduction of the image feature amount based on the basis space data of the basis space for image, 
 obtains basis space data of a basis space for text of which dimension is lower than the dimension of the text feature amount for mapping the text feature amount by using the text feature amount, and 
 performs the dimension reduction of the text feature amount based on the basis space data of the basis space for text. 
 
     
     
       10. The information processing device according to  claim 9 , wherein
 the learning means 
 obtains a code book used for vector quantization as the basis space data of the basis space for image by using the image feature amount, and 
 obtains a code representing a centroid vector as the image feature amount after the dimension reduction by performing the vector quantization of the image feature amount by using the code book. 
 
     
     
       11. The information processing device according to  claim 9 , wherein
 the learning means 
 extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals, 
 extracts a frequency of appearance of each word in the document as the text feature amount, 
 obtains a parameter of LDA (latent Dirichlet allocation) as the basis space data of the basis space for text by learning the LDA by using the document obtained from the learning content, and 
 converts the text feature amount obtained from the document to topic likelihood, which is likelihood of each latent topic of the LDA for the document, by using the parameter of the LDA, to obtain a topic label representing the latent topic of which topic likelihood is the maximum as the text feature amount after the dimension reduction. 
 
     
     
       12. The information processing device according to  claim 11 , wherein
 the learning means 
 generates a word dictionary of the words appearing in the document by using the document obtained from the learning content and creates a topic-to-frequently appearing word table of a word with high appearance frequency in the latent topic of the LDA and the appearance frequency of the word by using occurrence probability of occurrence of each word in the word dictionary in each latent topic of the LDA obtained by learning the LDA, and 
 the browsing controlling means 
 extracts the image feature amount of each frame of the image of the target content, performs the dimension reduction, and composes the annotation sequence by using the image feature amount after the dimension reduction, 
 obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model, 
 selects the latent topic represented by the topic label with high output probability in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence as a frame topic representing a content of the frame of interest, 
 selects a word with high appearance frequency in the frame topic as the annotation to be added to the frame of interest based on the topic-to-frequently appearing word table, 
 extracts a group of one or more temporally continuous frames to which the same annotation is added as the scene from the target content, and 
 displays the representative images of the scenes so as to be arranged in chronological order. 
 
     
     
       13. The information processing device according to  claim 12 , wherein
 the target content is content of a broadcast program, and 
 the browsing controlling means displays the representative images of the scenes of the broadcast program so as to be arranged in a program listing of the broadcast program on an EPG (electronic program guide) in chronological order. 
 
     
     
       14. The information processing device according to  claim 12 , wherein
 the browsing controlling means also displays the annotation added to the frame, which composes the scene, together with a representative image of the scene. 
 
     
     
       15. The information processing device according to  claim 12 , wherein
 the learning means 
 generates a word dictionary of the words appearing in the document by using the document obtained from the learning content and creates a topic-to-frequently appearing word table of a word with high appearance frequency in the latent topic of the LDA and the appearance frequency of the word by using occurrence probability of occurrence of each word in the word dictionary in each latent topic of the LDA obtained by learning the LDA, and 
 the browsing controlling means 
 extracts the image feature amount of each frame of the image of the target content, performs the dimension reduction, and composes the annotation sequence by using the image feature amount after the dimension reduction, 
 obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model, 
 selects the latent topic represented by the topic label with high output probability in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence as a frame topic representing a content of the frame of interest, 
 obtains an appearance frequency of a predetermined keyword in the frame topic based on the topic-to-frequently appearing word table and selects, when the appearance frequency of the predetermined keyword is high, the frame of interest as a keyword frame, which is a frame of which content coincides with the predetermined keyword, 
 extracts a group of one or more temporally continuous frames from the keyword frame as the scene, and 
 displays the representative images of the scenes so as to be arranged in chronological order. 
 
     
     
       16. The information processing device according to  claim 15 , wherein
 the target content is content of a broadcast program, and 
 the browsing controlling means displays the representative images of the scenes of the broadcast program so as to be arranged in chronological order in a program listing of the broadcast program on an EPG (electronic program guide). 
 
     
     
       17. An information processing method to be performed by an information processing device, comprising the steps of:
 extracting an image feature amount of each frame of an image of learning content and extracting word frequency information regarding a frequency of appearance of each word in a description text describing a content of the image of the learning content as a text feature amount of the description text,
 wherein the learning content includes a text of a caption, and 
 wherein the description text is the text of the caption included in the learning content; 
 
 learning an annotation models which is a multi-stream HMM (Hidden Markov Model), by using an annotation sequence for annotation, which is a multi-stream including the image feature amount and the text feature amount; 
 extracting a scene, which is a group of one or more temporally continuous frames, from target content from which the scene is to be extracted by using the annotation model; and 
 displaying representative images of scenes so as to be arranged in chronological order. 
 
     
     
       18. A non-transitory computer-readable medium having a set of computer-executable instructions embodied thereon to perform a method in a computing device comprising:
 extract an image feature amount of each frame of an image of learning content and extracting word frequency information regarding a frequency of appearance of each word in a description text describing a content of the image of the learning content as a text feature amount of the description text,
 wherein the learning content includes a text of a caption, and wherein the description text is the text of the caption included in the learning content; 
 learn an annotation model, which is a multi-stream HMM (Hidden Markov Model), by using an annotation sequence for annotation, which is a multi-stream including the image feature amount and the text feature amount; 
 
 extract a scene, which is a group of one or more temporally continuous frames, from target content from which the scene is to be extracted by using the annotation model; and 
 display representative images of scenes so as to be arranged in chronological order.