Autonomous video conferencing system with virtual director assistance

Abstract

Systems and methods are provided to power video conferencing and remote collaboration with subsymbolic and symbolic artificial intelligence. The autonomous video conferencing systems of this disclosure include one main smart camera and multiple peripheral smart cameras, optionally coupled with one or more smart sensors. Each smart camera is equipped with a vision pipeline supported by machine learning to detect objects and their interactions as well as related changes in gesture and posture, and a virtual director adapted to apply a predetermined rule set consistent with television studio production principles. The main camera is adapted to select and update a focus video stream in real time under the direction of its virtual director and stream the updated focus stream to a user computer. Methods for creating an automated television studio production for a variety of conferencing spaces and special-purpose scenarios with virtual director assistance are provided.

1 - 35 . (canceled) 
     
     
         36 . A camera system, comprising:
 at least one image sensor configured to capture an overview video stream; and   a video processing unit configured to:
 select a portion of the overview video stream to output as a focus video stream, wherein the focus video stream is selected based on one or more detected characteristics of a subject; and 
 cause the focus video stream to be shown on a display, 
 wherein the one or more detected characteristics include a location of eyes of the subject, and wherein the focus video streamed is framed such that the eyes of the subject are aligned with a separation between a top one third and a middle one third of a frame of the focus video stream. 
   
     
     
         37 . A camera system, comprising:
 at least one image sensor configured to capture an overview video stream including representations of at least a first subject and a second subject; and   a video processing unit configured to:
 automatically select a portion of the overview video stream to output as a focus video stream; and 
 cause the first focus video stream to be shown on a display, 
 wherein, if a representation of the first subject does not overlap with a representation of the second subject in the overview video stream, a first frame of the focus video stream is generated including a representation of the first subject free from a partial representation of the second subject, and a second frame of the focus video stream is generated including a representation of the second subject free from a partial representation of the first subject. 
   
     
     
         38 . The camera system of  claim 37 , wherein an overlap is present between the representation of the first subject and the representation of the second subject if a location of the representation of the second subject in the overview video stream causes the representation of the first subject in the overview video stream to be only partially visible. 
     
     
         39 . The camera system of  claim 37 , wherein the first frame and the second frame of the focus video stream are shown in succession. 
     
     
         40 . The camera system of  claim 37 , wherein the first subject is speaking. 
     
     
         41 . The camera system of  claim 37 , wherein, if a representation of the first subject overlaps with a representation of the second subject in the overview video stream, the focus video stream is generated to include at least one frame that includes the representation of the first subject and the representation of the second subject from the overview video stream. 
     
     
         42 . The camera system of  claim 37 , wherein the first subject is determined to be speaking. 
     
     
         43 . The camera system of  claim 37 , wherein the at least one image sensor includes a plurality of image sensors distributed among a corresponding plurality of cameras. 
     
     
         44 . The camera system of  claim 43 , wherein the video processing unit is included in one of the plurality of cameras. 
     
     
         45 . The camera system of  claim 37 , wherein the overview video stream includes representation of a meeting room, a workshop, or a classroom. 
     
     
         46 . A camera system, comprising:
 at least one image sensor configured to capture an overview video stream including a representation of at least one subject; and   a video processing unit configured to:
 select a portion of the overview video stream to output as a focus video stream, wherein the focus video stream features the at least one subject; and 
 cause the first focus video stream to be shown on a display, 
 wherein the focus video stream is framed to provide a first amount of frame space in a gaze direction of the subject that is greater than a second amount of frame space in a non-gaze direction of the subject. 
   
     
     
         47 . The camera system of  claim 46 , wherein the focus video stream also features a representation of an object in the gaze direction of the subject, and wherein the subject is a speaker. 
     
     
         48 . The camera system of  claim 46 , wherein the subject is a listener or is not speaking. 
     
     
         49 . A camera system, comprising:
 at least one image sensor configured to capture an overview video stream; and   a video processing unit configured to:
 select a portion of the overview video stream to output as a focus video stream, wherein the focus video stream is selected based on one or more detected characteristics of a non-speaking participant in a videoconference; and 
 cause the focus video stream to be shown on a display, wherein the focus video stream frames the non-speaking participant. 
   
     
     
         50 . The camera system of  claim 49 , wherein the one or more detected characteristics include a posture change or a gaze direction associated with the non-speaking participant 
     
     
         51 . The camera system of  claim 50 , wherein the posture change includes gesture performed by the non-speaking participant. 
     
     
         52 . The camera system of  claim 49 , wherein the focus video stream is shown on the display after a speaker shot or a presenter shot is displayed for a predefined length of time. 
     
     
         53 . The camera system of  claim 49 , wherein the video processing unit includes a hardware accelerated convolutional neural network. 
     
     
         54 . The camera system of  claim 49 , wherein the video processing unit and the at least one image sensor are located on a camera. 
     
     
         55 . The camera system of  claim 49 , wherein the focus video stream includes frame space padding in a gaze direction of the non-speaking participant. 
     
     
         56 . The camera system of  claim 49 , wherein the focus video stream includes representation of an object in a gaze direction of the non-speaking participant. 
     
     
         57 . A camera system, comprising:
 at least one image sensor configured to capture an overview video stream; and   a video processing unit configured to:
 automatically detect a videoconference participant and one or more objects with which the videoconference participant interacts; 
 select a portion of the overview video stream to output as a focus video stream, wherein the focus video stream is selected based on the detected videoconference participant and the one or more objects with which the videoconference participant interacts, and wherein the focus video stream is framed to feature both the videoconference participant and the one or more objects with which the videoconference participant interacts; and 
 cause the focus video stream to be shown on a display. 
   
     
     
         58 . The camera system of  claim 57 , wherein the one or more objects include a whiteboard. 
     
     
         59 . The camera system of  claim 57 , wherein the videoconference participant is a presenter determined to be speaking. 
     
     
         60 . A multi camera system, comprising:
 a plurality of image sensors distributed among a plurality of corresponding cameras, wherein at least one of the plurality of image sensors is configured to capture an overview video stream representative of a videoconference environment;   at least one microphone configured to provide direction of audio information; and   a video processing unit configured to:
 select a portion of the overview video stream to output as a focus video stream, wherein the focus video stream is selected based on a combination of the direction of audio information and one or more detected characteristics of a subject represented in the overview video stream; and 
 cause the focus video stream to be shown on a display. 
   
     
     
         61 . The camera system of  claim 60 , wherein the one or more detected characteristics include a location of eyes of the subject, and wherein the focus video streamed is framed such that the eyes of the subject are aligned with a separation between a top one third and a middle one third of a frame of the focus video stream. 
     
     
         62 . The camera system of  claim 60 , wherein the one or more detected characteristics include whether the subject is speaking. 
     
     
         63 . The camera system of  claim 60 , wherein the one or more detected characteristics include whether the subject is engaged in a gesture. 
     
     
         64 . The camera system of  claim 60 , wherein the one or more detected characteristics include a gaze direction of the subject. 
     
     
         65 . The camera system of  claim 60 , wherein the one or more detected characteristics include an extent of a visibility of the subject in a captured image frame. 
     
     
         66 . The camera system of  claim 60 , wherein the one or more detected characteristics include whether the subject is moving. 
     
     
         67 . The camera system of  claim 60 , wherein the at least one microphone is associated with one of the plurality of cameras. 
     
     
         68 . The camera system of  claim 60 , wherein the video processing unit is included in one of the plurality of cameras. 
     
     
         69 . The camera system of  claim 60 , wherein the video processing unit includes a hardware accelerated convolutional neural network.