Video stream segmentation for quality retention during reduced-bandwidth video conferencing

Abstract

During a video conference, a bandwidth reduction affecting a conference participant represented by a video stream without a virtual background is detected. Based on the bandwidth reduction, the video stream is segmented into a background and a foreground, in which the foreground depicts the conference participant. A resolution of the background is decreased while maintaining a resolution of the foreground. An adjusted video stream including the background at the decreased resolution and the foreground at the maintained resolution is then transmitted for rendering within a user interface of the video conference. Accordingly, portions of a video stream depicting a conference participant can be maintained at an initial quality level while adjusting other portions of the video stream to address network issues affecting a computing device of the conference participant.

What is claimed is: 
     
       1. A method, comprising:
 during a video conference, receiving, by a server device, a received video stream of a conference participant from a client device of the conference participant; 
 detecting, by the server device, a bandwidth reduction affecting the client device; 
 based on the bandwidth reduction, segmenting, by the server device, the received video stream into a background and a foreground, wherein the foreground depicts the conference participant; 
 selecting, by the server device, a single image from the background of one frame of the received video stream to use as the background for frames of the received video stream; 
 applying, by the server device, the single image as the background to the received video stream to obtain an adjusted video stream; and 
 transmitting, by the server device, the adjusted video stream for rendering within a user interface of the video conference. 
 
     
     
       2. The method of  claim 1 , comprising:
 restoring the received video stream for rendering within the user interface of the video conference based on a bandwidth of the client device associated with the conference participant returning to a threshold range. 
 
     
     
       3. The method of  claim 1 , wherein the bandwidth reduction is detected based on one or more of packet loss, round trip delay latency, or jitter affecting the received video stream. 
     
     
       4. The method of  claim 1 ,
 wherein the single image is a blurred background of the one frame of the received video stream. 
 
     
     
       5. A non-transitory computer readable medium storing instructions operable to cause one or more processors to perform operations comprising:
 during a video conference, receiving, by a server device, a received video stream of a conference participant from a client device of the conference participant; 
 detecting, by the server device, a bandwidth reduction affecting the client device; 
 based on the bandwidth reduction, segmenting, by the server device, the received video stream into a background and a foreground, wherein the foreground depicts the conference participant; 
 selecting, by the server device, a single image from the background of one frame of the received video stream to use as the background for frames of the received video stream; 
 applying, by the server device, the single image as the background to the received video stream to obtain an adjusted video stream; and 
 transmitting, by the server device, the adjusted video stream for rendering within a user interface of the video conference. 
 
     
     
       6. The non-transitory computer readable medium of  claim 5 , the operations comprising:
 detecting, during the video conference, that the bandwidth reduction has resolved; and 
 restoring the received video stream for rendering within the user interface of the video conference. 
 
     
     
       7. The non-transitory computer readable medium of  claim 5 , wherein the received video stream includes a top layer above the foreground, and wherein
 a resolution of the top layer is maintained during the bandwidth reduction. 
 
     
     
       8. The non-transitory computer readable medium of  claim 5 , wherein the operations for detecting the bandwidth reduction comprise:
 detecting that network information associated with the client device of the conference participant is below a threshold. 
 
     
     
       9. An apparatus, comprising:
 a memory; and 
 a processor configured to execute instructions stored in the memory to:
 during a video conference, receive, by a server device, a received video stream of a conference participant from a client device of the conference participant; 
 detect, by the server device, a bandwidth reduction affecting the client device; 
 based on the bandwidth reduction, segment, by the server device, the received video stream into a background and a foreground, wherein the foreground depicts the conference participant; 
 select, by the server device, a single image from the background of one frame of the received video stream to use as the background for frames of the received video stream; 
 apply, by the server device, the single image as the background to the received video stream to obtain an adjusted video stream; and 
 transmit, by the server device, the adjusted video stream for rendering within a user interface of the video conference. 
 
 
     
     
       10. The apparatus of  claim 9 , wherein a threshold used to detect the bandwidth reduction is selected based on an initial resolution of at least a portion of the received video stream. 
     
     
       11. The apparatus of  claim 9 , wherein the received video stream includes a top layer above the foreground, and wherein
 a resolution of the top layer is decreased during the bandwidth reduction. 
 
     
     
       12. The apparatus of  claim 9 , wherein
 the single image is a first image of the received video stream captured after detecting the bandwidth reduction. 
 
     
     
       13. The apparatus of  claim 9 , wherein the processor is configured to execute the instructions to:
 decrease a resolution of the background to a first decreased resolution; 
 determine that the background at the first decreased resolution fails to satisfy a bandwidth threshold; and 
 apply the single image as the background in response to determining that the bandwidth threshold is not satisfied. 
 
     
     
       14. The method of  claim 1 , wherein the single image is selected based on the single image having a lowest encoding throughput of a subset of evaluated frames of the received video stream. 
     
     
       15. The method of  claim 1 , wherein the single image is a first frame of the received video stream after the bandwidth reduction is detected. 
     
     
       16. The method of  claim 1 , wherein the single image is a key frame of the received video stream. 
     
     
       17. The method of  claim 1 , wherein the bandwidth reduction is received by the server device from the client device. 
     
     
       18. The method of  claim 1 , wherein the bandwidth reduction is detected by the server device. 
     
     
       19. The apparatus of  claim 9 , wherein the single image is selected based on the single image having a lowest encoding throughput of a subset of evaluated frames of the received video stream. 
     
     
       20. The apparatus of  claim 9 , wherein the single image is a first frame of the received video stream after the bandwidth reduction is detected.