System for say-feel gap analysis in video

Abstract

Systems and techniques using observed emotional data are described herein. An audio stream of a subject corresponding in time to a sequence of visual observations of the subject can be received. A transcript of speech uttered in the audio stream can be produced. A meaning of a string in the transcript can be determined. The sequence of visual observations that correspond to speech that produced the string can be received. An emotional state of the subject can be determined based on the sequence of visual observations. A correlation value can be calculated for the string by comparing the meaning and the emotional state.

What is claimed is: 
     
         1 . A system comprising:
 an audio processing module to:
 receive an audio stream of a subject corresponding in time to a sequence of visual observations of the subject; and 
 produce a transcript of speech uttered in the audio stream; 
   a semantic processing module to determine a meaning of a string in the transcript;   an image processing module to receive the sequence of visual observations that correspond to speech that produced the string;   an emotion determination module to determine an emotional state of the subject based on the sequence of visual observations; and   a difference module to calculate a correlation value for the string by comparing the meaning and the emotional state.   
     
     
         2 . The system of  claim 1  comprising a presentation module configured to present the correlation to a user. 
     
     
         3 . The system of  claim 2 , wherein to present the correlation includes the presentation module to play the sequence of visual observations, and to produce an audio representation with the sequence of visual observations. 
     
     
         4 . The system of  claim 3 , wherein the audio representation includes a modified aspect of the audio stream. 
     
     
         5 . The system of  claim 2 , wherein to present the correlation includes the presentation module to present a visual indication of the emotional state in a representation of the transcript corresponding to the string. 
     
     
         6 . The system of  claim 2 , wherein to present the correlation includes the presentation module to vary an intensity of the presentation based on the magnitude of the correlation. 
     
     
         7 . The system of  claim 1 , wherein the correlation includes an engagement component. 
     
     
         8 . The system of  claim 1 , wherein the correlation includes an emotional response component, the emotional response component including at least one of impact or appeal. 
     
     
         9 . A method comprising:
 receiving an audio stream of a subject corresponding in time to a sequence of visual observations of the subject;   producing a transcript of speech uttered in the audio stream;   determining a meaning of a string in the transcript;   receiving the sequence of visual observations that correspond to speech that produced the string;   determining an emotional state of the subject based on the sequence of visual observations; and   calculating a correlation value for the string by comparing the meaning and the emotional state.   
     
     
         10 . The method of  claim 9 , comprising presenting the correlation to a user. 
     
     
         11 . The method of  claim 10 , wherein presenting the correlation includes playing the sequence of visual observations, and producing an audio representation with the sequence of visual observations. 
     
     
         12 . The method of  claim 11 , wherein the audio representation includes a modified aspect of the audio stream. 
     
     
         13 . The method of  claim 10 , wherein presenting the correlation includes presenting a visual indication of the emotional state in a representation of the transcript corresponding to the string. 
     
     
         14 . The method of  claim 10 , wherein presenting the correlation includes creating a modified sequence of images by changing a portion of an image in the sequence of images, and playing the modified sequence of images. 
     
     
         15 . The method of  claim 10 , wherein presenting the correlation includes varying an intensity of the presentation based on the magnitude of the correlation. 
     
     
         16 . The method of  claim 9 , wherein the correlation includes an emotional response component, the emotional response component including at least one of impact or appeal. 
     
     
         17 . A machine readable medium that is not a transitory propagating signal, the machine readable medium including instruction that, when executed by a machine, cause the machine to perform operations comprising:
 receiving an audio stream of a subject corresponding in time to a sequence of visual observations of the subject;   producing a transcript of speech uttered in the audio stream;   determining a meaning of a string in the transcript;   receiving the sequence of visual observations that correspond to speech that produced the string;   determining an emotional state of the subject based on the sequence of visual observations; and   calculating a correlation value for the string by comparing the meaning and the emotional state.   
     
     
         18 . The machine readable medium of  claim 17 , wherein the operations include presenting the correlation to a user. 
     
     
         19 . The machine readable medium of  claim 18 , wherein presenting the correlation includes playing the sequence of visual observations, and producing an audio representation with the sequence of visual observations. 
     
     
         20 . The machine readable medium of  claim 19 , wherein the audio representation includes a modified aspect of the audio stream. 
     
     
         21 . The machine readable medium of  claim 18 , wherein presenting the correlation includes presenting a visual indication of the emotional state in a representation of the transcript corresponding to the string. 
     
     
         22 . The machine readable medium of  claim 18 , wherein presenting the correlation includes creating a modified sequence of images by changing a portion of an image in the sequence of images, and playing the modified sequence of images. 
     
     
         23 . The machine readable medium of  claim 18 , wherein presenting the correlation includes varying an intensity of the presentation based on the magnitude of the correlation. 
     
     
         24 . The machine readable medium of  claim 17 , wherein the correlation includes an emotional response component, the emotional response component including at least one of impact or appeal.