Multimodal identification and tracking of speakers in video

Abstract

A computer program product includes machine readable instructions for providing enhanced video output by: receiving footage including likeness information in a plurality of modalities; demultiplexing the plurality of modalities to provide information for each modality; comparing information from at least two of the modalities for determining a correlation in the likeness information; using the correlation, obtaining semantic information for association with the likeness; and combining the semantic information with the likeness information for providing the enhanced video output. A system for implementing the computer program product includes resources for receiving the footage.

1 . A computer program product comprising machine readable instructions stored on machine readable media, the instructions for providing enhanced video output by:
 receiving footage comprising likeness information in a plurality of modalities;   demultiplexing the plurality of modalities to provide information for each modality;   identifying a likeness using information from a first modality;   comparing the likeness with information from another modality and determining a correlation between the likeness and the information from another modality;   tracking the likeness using information from one of the first modality and the another modality;   obtaining semantic information for association with the likeness; and   associating the semantic information with the likeness for providing the enhanced video output.   
   
   
       2 . The computer program product of  claim 1 , wherein the plurality of modalities comprises at least one of a video modality, an audio modality, and a lip synchronization modality. 
   
   
       3 . The computer program product of  claim 1 , wherein the associating persists for a period of time that is one of longer and shorter than the correlation. 
   
   
       4 . The computer program product of  claim 1 , wherein the semantic information comprises at least one of personal information, a name, a player number, a political party and an employer identity. 
   
   
       5 . The computer program product of  claim 1 , wherein the likeness comprises appearance information comprising at least one of an auditory appearance and a visual appearance. 
   
   
       6 . The computer program product of  claim 1 , wherein at least one of the demultiplexing and the comparing comprises employing a speaker identification module comprising instructions for providing a speaker identification. 
   
   
       7 . The computer program product of  claim 1 , wherein tracking comprises employing a face detection and tracking module comprising instructions for providing at least one of face detection, facial recognition, face location determination, object tracking and image tracking. 
   
   
       8 . The computer program product of  claim 1 , wherein determining a correlation comprises employing a lip-sync detector module comprising instructions for detecting a spatio-temporal relationship between voice and motion in the footage. 
   
   
       9 . A system for providing enhanced video output, the system comprising:
 resources for receiving footage, and for implementing a computer program product comprising machine readable instructions stored on machine readable media, the instructions for providing enhanced video output by:   receiving footage comprising likeness information in a plurality of modalities;   demultiplexing the plurality of modalities to provide information for each modality;   identifying a likeness using information from a first modality;   comparing the likeness with information from another modality and determining a correlation between the likeness and the information from another modality;   tracking the likeness using information from one of the first modality and the another modality;   obtaining semantic information for association with the likeness; and   associating the semantic information with the likeness for providing the enhanced video output.   
   
   
       10 . A system for providing enhanced video output, the system comprising:
 resources for receiving footage, and for implementing a computer program product comprising machine readable instructions stored on machine readable media, the instructions for providing enhanced video output by:   receiving footage comprising likeness information in a plurality of modalities;   demultiplexing the plurality of modalities to provide information for each modality;   comparing information from at least two of the modalities for determining a correlation in the likeness information;   using the correlation, obtaining semantic information for association with the likeness; and   combining the semantic information with the likeness information for providing the enhanced video output.   
   
   
       11 . A computer program product comprising machine readable instructions stored on machine readable media, the instructions for providing enhanced video output by:
 receiving footage comprising likeness information in a plurality of modalities comprising at least one of a video modality, an audio modality, and a lip synchronization modality, the likeness information comprising appearance information comprising at least one of an auditory appearance and a visual appearance;   demultiplexing the plurality of modalities to provide information for each modality;   identifying a likeness using information from a first modality;   comparing the likeness with information from another modality and determining a correlation between the likeness and the information from another modality;   tracking the likeness using information from one of the first modality and the another modality;   associating the semantic information with the likeness for providing the enhanced video output;   wherein at least one of the demultiplexing and the comparing comprises employing at least one of a speaker identification module comprising instructions for providing a speaker identification, a shot boundary detection module comprising instructions for providing at least one of a start time and an end time for the speaker, a face detection and tracking module comprising instructions for providing at least one of facial recognition, location determination and image tracking, and a lip-sync detector module comprising instructions for detecting a spatio-temporal relationship between voice and motion in the footage;   obtaining semantic information for association with the likeness; the semantic information comprising at least one of personal information, a name, a player number, a political party and an employer identity for association with the likeness; and   associating the semantic information with the likeness for providing the enhanced video output for a period of time that is one of longer and shorter than the correlation.