Multi-modal modeling of temporal interaction sequences

Abstract

A multi-modal interaction modeling system can model a number of different aspects of a human interaction across one or more temporal interaction sequences. Some versions of the system can generate assessments of the nature or quality of the interaction or portions thereof, which can be used to, among other things, provide assistance to one or more of the participants in the interaction.

The invention claimed is: 
     
       1. A method for assessing an interaction involving at least two participants, at least one of the participants being a person, the method comprising, with a computing system:
 detecting, from multi-modal data captured by at least one sensing device during the interaction, a plurality of different behavioral cues expressed by the participants; 
 analyzing the detected behavioral cues with respect to a plurality of different time scales, each of the time scales being defined by a time interval whose size is compared to the size of other time intervals of the interaction, wherein the plurality of different time scales comprise at least two of a short term time scale, a medium term time scale, and a long term time scale; 
 recognizing, using machine learning and based on the analysis of the detected behavioral cues, a temporal interaction sequence comprising a pattern of the behavioral cues corresponding to one or more of the time scales; and 
 deriving, from the temporal interaction sequence, an assessment of the efficacy of the interaction; 
 wherein at least one indication of the assessment is provided through a device to a user or an application of the computing system. 
 
     
     
       2. The method of  claim 1 , wherein the plurality of different behavioral cues of the temporal interaction sequence involves at least one non-verbal cue. 
     
     
       3. The method of  claim 1 , comprising recognizing a plurality of temporal interaction sequences occurring over different time intervals, at least one of the temporal interaction sequences involving behavioral cues expressed by the person and behavioral cues expressed by another participant, and deriving the assessment from the plurality of temporal interaction sequences. 
     
     
       4. The method of  claim 1 , wherein each of the plurality of different behavioral cues comprises a cue relating to one or more of: a gesture, a body pose, a pose, an eye gaze, a facial expression, a voice tone, a voice loudness, another non-verbal vocal feature, and verbal content. 
     
     
       5. The method of  claim 1 , comprising semantically analyzing verbal content of at least one of the behavioral cues and recognizing the temporal interaction sequence based on the semantic analysis of the verbal content. 
     
     
       6. The method of  claim 1 , comprising semantically analyzing the verbal content of at least one of the behavioral cues and deriving the assessment based on the semantic analysis of the verbal content. 
     
     
       7. The method of  claim 1 , comprising analyzing the relative significance of each behavioral cue in relation to the temporal interaction sequence as a whole, and deriving the assessment based on the relative significance of each behavioral cue to the temporal interaction sequence as a whole. 
     
     
       8. The method of  claim 1 , comprising analyzing the relative significance of each behavioral cue in relation to the other behavioral cues of the temporal interaction sequence, and deriving the assessment based on the relative significance of each of the behavioral cues of the temporal interaction sequence. 
     
     
       9. The method of  claim 1 , comprising recognizing a plurality of temporal interaction sequences, deriving an assessment of the nature of each temporal interaction sequence, and determining changes in the efficacy of the interaction over time based on the assessments of each of the temporal interaction sequences. 
     
     
       10. The method of  claim 1 , comprising recognizing a plurality of temporal interaction sequences, deriving an assessment of the nature of each temporal interaction sequence, and evaluating the overall efficacy of the interaction based on the assessments of the temporal interaction sequences. 
     
     
       11. The method of  claim 1 , comprising recognizing a plurality of temporal interaction sequences having overlapping time intervals and deriving the assessment based on the plurality of temporal interaction sequences having overlapping time intervals. 
     
     
       12. The method of  claim 1 , wherein the plurality of behavioral cues of the temporal interaction sequence comprises at least one behavioral cue that indicates a positive interaction and at least one behavioral cue that indicates a negative interaction. 
     
     
       13. The method of  claim 1 , comprising presenting a suggestion to one or more of the participants based on the assessment. 
     
     
       14. The method of  claim 1 , comprising generating a description of the interaction based on the assessment. 
     
     
       15. The method of  claim 14 , wherein the description comprises a recounting of the interaction, and the recounting comprises one or more behavioral cues having evidentiary value to the assessment. 
     
     
       16. The method of  claim 1 , comprising determining a semantic meaning of the pattern of behavioral cues and using the semantic meaning to derive the assessment. 
     
     
       17. The method of  claim 1 , comprising determining the pattern of behavioral cues using a graphical model. 
     
     
       18. The method of  claim 17 , wherein the graphical model comprises a discriminative probabilistic model. 
     
     
       19. The method of  claim 18 , wherein the discriminative probabilistic model comprises conditional random fields. 
     
     
       20. The method of  claim 1 , comprising developing the assessment of the efficacy of the interaction based on changes in the behavioral cues of at least one of the participants over time during the interaction. 
     
     
       21. The method of  claim 1 , comprising inferring one or more events from the temporal interaction sequence, the one or more events each comprising a semantic characterization of at least a portion of the temporal interaction sequence, deriving an assessment of the one or more events based on the behavioral cues, and deriving the assessment of the efficacy of the interaction based on the assessment of the one or more events. 
     
     
       22. The method of  claim 1 , comprising determining relationships between or among the behavioral cues of the temporal interaction sequence. 
     
     
       23. The method of  claim 1 , wherein the duration of each of the plurality of different time scales is defined by the detection of at least two of the behavioral cues. 
     
     
       24. A method for assessing an interaction involving at least two participants, at least one of the participants being a person, the method comprising, with a computing system:
 detecting, from multi-modal data captured by at least one sensing device, a plurality of different behavioral cues expressed by the participants during the interaction, the behavioral cues comprising one or more non-verbal cues and verbal content; 
 recognizing, using machine learning, a temporal interaction sequence comprising a pattern of the behavioral cues occurring over a time interval during the interaction, wherein the time interval corresponds to at least one time scale, the at least one time scale being one of a short term time scale, a medium term time scale, and a long term time scale; and 
 deriving, from the temporal interaction sequence, an assessment of the efficacy of the interaction; 
 wherein at least one indication of the assessment is provided through a device to a user or an application of the computing system.