Systems and methods for audience interactions in real-time multimedia applications

Abstract

Systems and methods for audience interaction in real-time multimedia applications are provided. In one embodiment, a method for a real-time video conference comprises, during the real-time video conference, receiving a media stream including audio and video from a remote computing system over a network, acquiring, video and audio of a user, detecting an event with a machine learning model in at least one of the video and the audio of the user, and transmitting an event detection message indicating the detected event and/or audio and video of the user to the remote computing system over the network. In this way, natural audience reactions may be automatically detected and shared.

1 . A method for a real-time video conference, comprising:
 receiving, at a computing system from a remote computing system over a network, a media stream including audio and video;   acquiring, via a camera and a microphone of the computing system, video and audio of a user of the computing system;   detecting, with a machine learning model, an event in at least one of the video and the audio of the user; and   transmitting, to the remote computing system over the network, an event detection message indicating the detected event.   
     
     
         2 . The method of  claim 1 , further comprising:
 detecting, with the machine learning model, an end of the event in at least one of the video and the audio of the user; and   transmitting, to the remote computing system over the network, an event detection message indicating the detected end of the event.   
     
     
         3 . The method of  claim 1 , further comprising selectively transmitting, to the remote computing system over the network, at least one of the video and the audio of the user including the event responsive to detecting the event. 
     
     
         4 . The method of  claim 1 , wherein the media stream further comprises audio and video of a second user from a second computing system. 
     
     
         5 . The method of  claim 4 , further comprising transmitting, to the second computing system over the network, the video and the audio of the user. 
     
     
         6 . The method of  claim 1 , wherein the event comprises at least one of clapping or cheering by the user, and wherein the event is detected by the machine learning model in the audio of the user. 
     
     
         7 . A method for a real-time video conference, comprising:
 acquiring, via a camera and a microphone of a computing system, video and audio of a user of the computing system;   transmitting, to a remote computing system over a network, the video and the audio of the user;   receiving, from the remote computing system over the network, a media stream and an event detection message indicating an event automatically detected with a machine learning model at the remote computing system in the media stream; and   automatically outputting, via an output device of the computing system, the media stream based on the event detection message.   
     
     
         8 . The method of  claim 7 , further comprising:
 receiving, from the remote computing system over the network, an additional event detection message indicating an end of the event automatically detected with the machine learning model in the media stream; and   automatically ceasing output of the media stream via the output device responsive to the additional event detection message.   
     
     
         9 . The method of  claim 7 , wherein the media stream comprises audio of a remote user of the remote computing system, and wherein automatically outputting the media stream comprises automatically outputting the audio of the remote user to at least one speaker of the computing system. 
     
     
         10 . The method of  claim 7 , wherein the media stream comprises video of a remote user of the remote computing system, and wherein automatically outputting the media stream comprises automatically outputting the video of the remote user to at least one display device of the computing system. 
     
     
         11 . The method of  claim 7 , further comprising:
 transmitting, to a second remote computing system over the network, the video and the audio of the user;   receiving, from the second remote computing system over the network, a second media stream and a second event detection message indicating a second event automatically detected with a second machine learning model at the second remote computing system in the second media stream;   mixing the media stream and the second media stream; and   automatically outputting, via the output device of the computing system, the mixed media stream.   
     
     
         12 . A system for a real-time video conference, comprising:
 a plurality of computing systems including at least a first computing system and a second computing system communicatively coupled via a network and configured to acquire a first media stream and a second media stream, respectively;   wherein the first computing system is configured with executable instructions in non-transitory memory that when executed cause a processor of the first computing system to:
 automatically detect, with a machine learning model, an event in the first media stream, the first media stream comprising one or more of audio and video of a user of the first computing system; and 
 transmit, to the second computing system over the network, an event detection message indicating the event; 
   wherein the second computing system is configured with executable instructions in non-transitory memory that when executed cause a processor of the second computing system to:
 receive, from the first computing system over the network, the event detection message; and 
 control an output device of the second computing system to indicate the event to a user of the second computing system responsive to receiving the event detection message. 
   
     
     
         13 . The system of  claim 12 , wherein the output device of the second computing system comprises a display device, and wherein the second computing system is configured with executable instructions in non-transitory memory that when executed cause the processor of the second computing system to control the display device to display a visual indicator of the event, the visual indicator dynamically adjusted based on the event detection message. 
     
     
         14 . The system of  claim 12 , wherein the output device of the second computing system comprises an audio output device, and wherein the second computing system is configured with executable instructions in non-transitory memory that when executed cause the processor of the second computing system to control the audio output device to output an indication of the event, the indication of the event comprising one of a synthesized audio output or a portion of the first media stream including the event. 
     
     
         15 . The system of  claim 12 , wherein the plurality of computing systems further includes a third computing system communicatively coupled to the first computing system and the second computing system via the network and configured to acquire a third media stream. 
     
     
         16 . The system of  claim 15 , wherein the third computing system is configured with executable instructions in non-transitory memory that when executed cause a processor of the third computing system to:
 automatically detect, with a machine learning model, a second event in the third media stream, the third media stream comprising one or more of audio and video of a user of the third computing system; and   transmit, to the second computing system over the network, a second event detection message indicating the second event;   wherein the second computing system is configured with executable instructions in non-transitory memory that when executed cause the processor of the second computing system to:   receive, from the third computing system over the network, the second event detection message; and   control the output device of the second computing system to indicate the second event to the user of the second computing system responsive to receiving the second event detection message.   
     
     
         17 . The system of  claim 16 , wherein the output device comprises a plurality of speakers including a first speaker and a second speaker, and wherein the second computing system is configured with executable instructions in non-transitory memory that when executed cause the processor of the second computing system to:
 control the first speaker to output an auditory indication of the event and the second speaker to output an auditory indication of the second event.   
     
     
         18 . The system of  claim 16 , wherein the first computing system and the third computing system receive the third media stream and the first media stream, respectively, over the network with the second media stream. 
     
     
         19 . The system of  claim 12 , wherein the machine learning model comprises a deep neural network executed in a browser of the first computing system. 
     
     
         20 . The system of  claim 12 , wherein the event comprises one or more of clapping, cheering, and laughing, and wherein the machine learning model detects the event in the audio of the user of the first computing system.