Real-Time Recommendation Based On Audience Sentiment

Abstract

Video data from audience participants reacting to a speaker participation during a conference is obtained. The video data is processed to detect and recognize reactions based on a speaker presentation. Sentiment types are determined for the recognized reactions in view of a context of the speaker presentation. An engagement level is determined based on aggregated sentiment types for the audience participants. A real-time recommendation output is presented based on the engagement level. The real-time recommendation output provides suggestive actions for the speaker participant based on a positive or negative engagement level.

What is claimed is: 
     
         1 . A method, comprising: 
 aggregating, by a server during multiple previous video conference sessions, previous conference session information including previous speaker participant behaviors and previous engagement levels;   determining, by the server during a video conference, sentiment types based on reactions of one or more audience participants to speaker participant behaviors of a speaker participant;   determining, by the server during the video conference, a first engagement level based on the sentiment types;   determining, by the server during the video conference using a machine learning model based in part on the previous conference session information, a real-time recommendation based on the first engagement level and the speaker participant behaviors;   outputting, by the server during the video conference, the real-time recommendation to a device associated with the speaker participant to allow the speaker participant to change the speaker participant behaviors during the video conference;   determining, by the server after outputting the real-time recommendation, a second engagement level based on reactions of the one or more audience participants; and   maintaining, by the server, engagement trends based on a change between the first engagement level and the second engagement level.   
     
     
         2 . The method of  claim 1 , wherein the previous conference session information further includes timestamps for the previous speaker participant behaviors and the previous engagement levels. 
     
     
         3 . The method of  claim 1 , wherein the previous conference session information further includes reaction detections, the sentiment types, engagement levels, and real-time recommendation outputs. 
     
     
         4 . The method of  claim 1 , wherein the real-time recommendation comprises at least one of an indication to maintain a speaker participant behavior, an indication to change the speaker participant behavior, an indication to maintain a current presentation topic, an indication to change to a different presentation topic, or an indication to pause the video conference for questions. 
     
     
         5 . The method of  claim 1 , wherein determining the sentiment types comprises using facial recognition and movement detection on video data of the one or more audience participants. 
     
     
         6 . The method of  claim 1 , wherein determining the sentiment types further comprises detecting, by the server, verbal reactions of the one or more audience participants from a real-time transcription of audio data of the video conference. 
     
     
         7 . The method of  claim 1 , further comprising generating, by the server, a real-time transcription of audio data of the video conference, determining, by the server, a context of the video conference by using a contextual machine learning model to the real-time transcription, and determining, by the server, the sentiment types in view of the context of the video conference such that at least one reaction is assigned a first sentiment type when the context indicates a first context type and is assigned a second sentiment type different from the first sentiment type when the context indicates a second context type. 
     
     
         8 . The method of  claim 1 , further comprising: 
 maintaining a histogram of the sentiment types, wherein the histogram comprises bins for different ones of the sentiment types, wherein the first engagement level and the second engagement level are determined based on a most frequently occurring sentiment type in the histogram.   
     
     
         9 . The method of  claim 1 , further comprising: assigning numeric values from a range of values to each of the sentiment types, wherein the first engagement level and the second engagement level are determined based on a total of the numeric values. 
     
     
         10 . The method of  claim 1 , wherein outputting the real-time recommendation comprises transmitting the real-time recommendation to a secondary device associated with the speaker participant. 
     
     
         11 . The method of  claim 10 , wherein the secondary device comprises at least one of a tablet, a wearable device, or a lectern device. 
     
     
         12 . The method of  claim 10 , wherein the real-time recommendation is output on the secondary device while a gallery view displayed on a primary device of the speaker participant is uninterrupted. 
     
     
         13 . An apparatus, comprising: 
 a memory; and    a processor configured to execute instructions stored in the memory to:      aggregate, during multiple previous video conference sessions, previous conference session information including previous speaker participant behaviors and previous engagement levels;    determine, during a video conference, sentiment types based on reactions of one or more audience participants to speaker participant behaviors of a speaker participant;    determine, during the video conference, a first engagement level based on the sentiment types;    determine, during the video conference using a machine learning model based in part on the previous conference session information, a real-time recommendation based on the first engagement level and the speaker participant behaviors;    output, during the video conference, the real-time recommendation to a device associated with the speaker participant to allow the speaker participant to change the speaker participant behaviors during the video conference;    determine, after outputting the real-time recommendation, a second engagement level based on reactions of the one or more audience participants; and    maintain engagement trends based on a change between the first engagement level and the second engagement level.     
     
     
         14 . The apparatus of  claim 13 , wherein the previous conference session information further includes reaction detections, the sentiment types, engagement levels, real-time recommendation outputs, and timestamps for the previous speaker participant behaviors and the previous engagement levels.  
     
     
         15 . The apparatus of  claim 13 , wherein the processor is configured to execute the instructions to maintain a histogram of the sentiment types, wherein the histogram comprises bins for different ones of the sentiment types, wherein the first engagement level and the second engagement level are determined based on a most frequently occurring sentiment type in the histogram.  
     
     
         16 . The apparatus of  claim 13 , wherein the processor is configured to execute the instructions to assign numeric values from a range of values to each of the sentiment types, wherein the first engagement level and the second engagement level are determined based on a total of the numeric values. 
     
     
         17 . A non-transitory computer-readable medium storing instructions operable to cause one or more processors to perform operations comprising: 
 aggregating, during multiple previous video conference sessions, previous conference session information including previous speaker participant behaviors and previous engagement levels;   determining, during a video conference, sentiment types based on reactions of one or more audience participants to speaker participant behaviors of a speaker participant;   determining, during the video conference, a first engagement level based on the sentiment types;   determining, during the video conference using a machine learning model based in part on the previous conference session information, a real-time recommendation based on the first engagement level and the speaker participant behaviors;   outputting, during the video conference, the real-time recommendation to a device associated with the speaker participant to allow the speaker participant to change the speaker participant behaviors during the video conference;   determining, after outputting the real-time recommendation, a second engagement level based on reactions of the one or more audience participants; and   maintaining engagement trends based on a change between the first engagement level and the second engagement level.   
     
     
         18 . The non-transitory computer-readable medium of  claim 17 , wherein the previous conference session information further includes reaction detections, the sentiment types, engagement levels, real-time recommendation outputs, and timestamps for the previous speaker participant behaviors and the previous engagement levels.  
     
     
         19 . The non-transitory computer-readable medium of  claim 17 , the operations further comprising: 
 maintaining a histogram of the sentiment types, wherein the histogram comprises bins for different ones of the sentiment types, wherein the first engagement level and the second engagement level are determined based on a most frequently occurring sentiment type in the histogram.   
     
     
         20 . The non-transitory computer-readable medium of  claim 17 , the operations further comprising: 
 assigning numeric values from a range of values to each of the sentiment types, wherein the first engagement level and the second engagement level are determined based on a total of the numeric values.