Alignment of video and textual sequences for metadata analysis

Abstract

Systems, methods and computer program products related to aligning heterogeneous sequential data are disclosed. Video data in a media presentation and textual data corresponding to content of the media presentation are received. An action related to aligning the video data and the textual data is determined using an alignment neural network, such that the video data and the textual data are at least partially aligned following the action. The alignment neural network includes a first fully connected layer that receives as input the video data, the textual data, and data relating to a previously determined action by the alignment neural network related to aligning the video data and the textual data. The determined action related to aligning the video data and the textual data is performed.

What is claimed is: 
     
       1. A method comprising:
 receiving video data in a media presentation and textual data corresponding to content of the media presentation at an alignment neural network previously trained to align training video data and training textual data; 
 determining a first action related to aligning the video data and the textual data using the trained alignment neural network, such that the video data and the textual data are at least partially aligned following the first action, the trained alignment neural network comprising:
 a first fully connected layer that receives as input:
 the video data, 
 the textual data, and 
 data relating to a previously determined action by the trained alignment neural network related to aligning the video data and the textual data; and 
 
 
 performing the first action related to aligning the video data and the textual data. 
 
     
     
       2. The method of  claim 1 , wherein the data relating to the previously determined action by the trained alignment neural network comprises:
 data relating to a previously determined match between the video data and the textual data. 
 
     
     
       3. The method of  claim 1 , wherein the trained alignment neural network further comprises a second fully connected layer that receives as input data from the first fully connected layer. 
     
     
       4. The method of  claim 1 , further comprising storing data related to the first action. 
     
     
       5. The method of  claim 1 , wherein the first action comprises at least one of: a pop action related to the video data, a pop action related to the textual data, a match action, or a match-retain action. 
     
     
       6. The method of  claim 1 , wherein the first action comprises a match action. 
     
     
       7. The method of  claim 1 , further comprising:
 receiving audio data, wherein the first action relates to aligning the video data, the textual data, and the audio data. 
 
     
     
       8. The method of  claim 7 , wherein the first action is a parameterized action. 
     
     
       9. The method of  claim 1 , wherein the video data comprises a feature vector generated based on mean pooling a plurality of video frames. 
     
     
       10. A computer program product, comprising:
 a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured to perform one or more operations comprising:
 receiving video data in a media presentation and textual data corresponding to content of the media presentation at an alignment neural network previously trained to align training video data and training textual data; 
 determining a first action related to aligning the video data and the textual data using the trained alignment neural network, such that the video data and the textual data are at least partially aligned following the first action, the trained alignment neural network comprising:
 a first fully connected layer that receives as input:
 the video data, 
 the textual data, and 
 data relating to a previously determined action by the trained alignment neural network related to aligning the video data and the textual data; and 
 
 
 performing the first action related to aligning the video data and the textual data. 
 
 
     
     
       11. The computer program product of  claim 10 , wherein the data relating to the previously determined action by the trained alignment neural network comprises
 data relating to a previously determined match between the video data and the textual data. 
 
     
     
       12. The computer program product of  claim 10 , wherein the trained alignment neural network further comprises a second fully connected layer that receives as input data from the first fully connected layer. 
     
     
       13. The computer program product of  claim 10 , wherein the first action comprises at least one of: a pop action related to the video data, a pop action related to the textual data, a match action, or a match-retain action. 
     
     
       14. The computer program product of  claim 10 , wherein the first action comprises a match action. 
     
     
       15. The computer program product of  claim 10 , the operations further comprising:
 receiving audio data, wherein the first action relates to aligning the video data, the textual data, and the audio data, and wherein the first action is a parameterized action. 
 
     
     
       16. A system, comprising:
 a processor; and 
 a memory containing a program that, when executed on the processor, performs one or more operations comprising:
 receiving video data in a media presentation and textual data corresponding to content of the media presentation at an alignment neural network previously trained to align video data and textual data; 
 determining a first action related to aligning the video data and the textual data using the trained alignment neural network, such that the video data and the textual data are at least partially aligned following the first action, the trained alignment neural network comprising:
 a first fully connected layer that receives as input:
 the video data, 
 the textual data, and 
 data relating to a previously determined action by the trained alignment neural network related to aligning the video data and the textual data; and 
 
 
 performing the first action related to aligning the video data and the textual data. 
 
 
     
     
       17. The system of  claim 16 , wherein the data relating to the previously determined action by the trained alignment neural network comprises:
 data relating to a previously determined match between the video data and the textual data. 
 
     
     
       18. The system of  claim 16 , wherein the trained alignment neural network further comprises a second fully connected layer that receives as input data from the first fully connected layer. 
     
     
       19. The system of  claim 16 , wherein the first action comprises a match action. 
     
     
       20. The system of  claim 16 , the operations further comprising:
 receiving audio data, wherein the first action relates to aligning the video data, the textual data, and the audio data, and wherein the first action is a parameterized action.