Real-time video enhancement

Abstract

Methods and systems for real-time video enhancement are provided herein. A current frame of a video stream generated by a client device of a plurality of client devices participating in the video conference is identified during a video conference. An enhanced previous frame corresponding to an enhanced version of a previous frame in the video stream is identified. At least the current frame and the enhanced previous frame are provided as input to a machine-learning model. An output of the machine learning model is obtained. The output of the machine learning model indicates an enhanced current frame corresponding to an enhanced version of the current frame. The current frame is replaced with the enhanced current frame in the video stream.

What is claimed is: 
     
         1 . A method comprising:
 identifying, during a video conference, a current frame of a video stream generated by a client device of a plurality of client devices participating in the video conference;   identifying, by a processing device, an enhanced previous frame corresponding to an enhanced version of a previous frame in the video stream;   providing, by the processing device, at least the current frame and the enhanced previous frame as input to a machine learning model;   obtaining, by the processing device, an output of the machine learning model, the output of the machine learning model indicating an enhanced current frame corresponding to an enhanced version of the current frame; and   causing the current frame to be replaced, in the video stream, with the enhanced current frame.   
     
     
         2 . The method of  claim 1 , wherein obtaining the output of the machine learning model comprises:
 obtaining, using a first subnetwork of one or more layers of the machine learning model, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame;   obtaining, using a second subnetwork of one or more layers of the machine learning model, a first set of encoded tokens and a second set of encoded tokens based on the first set of tokens and the second set of tokens; and   obtaining, using a third subnetwork of one or more layers of the machine learning model, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens.   
     
     
         3 . The method of  claim 2 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:
 generating, using a convolutional neural network of the first subnetwork, the first set of tokens containing spatial and temporal information associated with a combination of the current frame and the enhanced previous frame.   
     
     
         4 . The method of  claim 2 , wherein obtaining, using the second subnetwork, a first set of encoded tokens and a second set of encoded tokens comprises:
 generating, using a first mixer layer of the second subnetwork, the first set of encoded tokens based on encoding the first set of tokens with learned spatio-temporal features; and   generating, using a second mixer layer of the second subnetwork, the second set of encoded tokens based on encoding the second set of tokens with learned temporal features.   
     
     
         5 . The method of  claim 2 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:
 generating, using a first linear layer of the first subnetwork, a first set of intermediary tokens containing temporal information associated with the current frame;   generating, using a second linear layer of the first subnetwork, a second set of intermediary tokens containing temporal information associated with the previous frame; and   combining the first set of intermediary tokens and the second set of intermediary tokens into the second set of tokens.   
     
     
         6 . The method of  claim 2 , wherein obtaining, using the third subnetwork, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens comprises:
 combining the first set of encoded tokens and the second set of encoded tokens; and   generating, using a decoder of the third subnetwork, the enhanced current frame based on the combination of the first set of encoded tokens and the second set of encoded tokens.   
     
     
         7 . The method of  claim 1 , wherein the machine learning model is trained to perform at least one of: deblurring, denoising, or super-resolution to correct a low quality of a frame. 
     
     
         8 . A system comprising:
 a memory device; and   a processing device coupled to the memory device, wherein the processing device is to perform operations comprising:
 identifying, during a video conference, a current frame of a video stream generated by a client device of a plurality of client devices participating in the video conference; 
 identifying, by a processing device, an enhanced previous frame corresponding to an enhanced version of a previous frame in the video stream; 
 providing, by the processing device, at least the current frame and the enhanced previous frame as input to a machine learning model; 
 obtaining, by the processing device, an output of the machine learning model, the output of the machine learning model indicating an enhanced current frame corresponding to an enhanced version of the current frame; and 
 causing the current frame to be replaced, in the video stream, with the enhanced current frame. 
   
     
     
         9 . The system of  claim 8 , wherein obtaining the output of the machine learning model comprises:
 obtaining, using a first subnetwork of one or more layers of the machine learning model, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame;   obtaining, using a second subnetwork of one or more layers of the machine learning model, a first set of encoded tokens and a second set of encoded tokens based on the first set of tokens and the second set of tokens; and   obtaining, using a third subnetwork of one or more layers of the machine learning model, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens.   
     
     
         10 . The system of  claim 9 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:
 generating, using a convolutional neural network of the first subnetwork, the first set of tokens containing spatial and temporal information associated with a combination of the current frame and the enhanced previous frame.   
     
     
         11 . The system of  claim 9 , wherein obtaining, using the second subnetwork, a first set of encoded tokens and a second set of encoded tokens comprises:
 generating, using a first mixer layer of the second subnetwork, the first set of encoded tokens based on encoding the first set of tokens with learned spatio-temporal features; and   generating, using a second mixer layer of the second subnetwork, the second set of encoded tokens based on encoding the second set of tokens with learned temporal features.   
     
     
         12 . The system of  claim 9 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:
 generating, using a first linear layer of the first subnetwork, a first set of intermediary tokens containing temporal information associated with the current frame;   generating, using a second linear layer of the first subnetwork, a second set of intermediary tokens containing temporal information associated with the previous frame; and   combining the first set of intermediary tokens and the second set of intermediary tokens into the second set of tokens.   
     
     
         13 . The system of  claim 9 , wherein obtaining, using the third subnetwork, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens comprises:
 combining the first set of encoded tokens and the second set of encoded tokens; and   generating, using a decoder of the third subnetwork, the enhanced current frame based on the combination of the first set of encoded tokens and the second set of encoded tokens.   
     
     
         14 . The system of  claim 8 , wherein the machine learning model is trained to perform at least one of: deblurring, denoising, or super-resolution to correct a low quality of a frame. 
     
     
         15 . A non-transitory computer-readable medium comprising instructions that, responsive to execution by a processing device, cause the processing device to perform operations comprising:
 identifying, during a video conference, a current frame of a video stream generated by a client device of a plurality of client devices participating in the video conference;   identifying, by a processing device, an enhanced previous frame corresponding to an enhanced version of a previous frame in the video stream   providing, by the processing device, at least the current frame and the enhanced previous frame as input to a machine learning model;   obtaining, by the processing device, an output of the machine learning model, the output of the machine learning model indicating an enhanced current frame corresponding to an enhanced version of the current frame; and   causing the current frame to be replaced, in the video stream, with the enhanced current frame.   
     
     
         16 . The non-transitory computer-readable medium of  claim 15 , wherein obtaining the output of the machine learning model comprises:
 obtaining, using a first subnetwork of one or more layers of the machine learning model, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame;   obtaining, using a second subnetwork of one or more layers of the machine learning model, a first set of encoded tokens and a second set of encoded tokens based on the first set of tokens and the second set of tokens; and   obtaining, using a third subnetwork of one or more layers of the machine learning model, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens.   
     
     
         17 . The non-transitory computer-readable medium of  claim 16 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:
 generating, using a convolutional neural network of the first subnetwork, the first set of tokens containing spatial and temporal information associated with a combination of the current frame and the enhanced previous frame.   
     
     
         18 . The non-transitory computer-readable medium of  claim 16 , wherein obtaining, using the second subnetwork, a first set of encoded tokens and a second set of encoded tokens comprises:
 generating, using a first mixer layer of the second subnetwork, the first set of encoded tokens based on encoding the first set of tokens with learned spatio-temporal features; and   generating, using a second mixer layer of the second subnetwork, the second set of encoded tokens based on encoding the second set of tokens with learned temporal features.   
     
     
         19 . The non-transitory computer-readable medium of  claim 16 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:
 generating, using a first linear layer of the first subnetwork, a first set of intermediary tokens containing temporal information associated with the current frame;   generating, using a second linear layer of the first subnetwork, a second set of intermediary tokens containing temporal information associated with the previous frame; and   combining the first set of intermediary tokens and the second set of intermediary tokens into the second set of tokens.   
     
     
         20 . The non-transitory computer-readable medium of  claim 16 , wherein obtaining, using the third subnetwork, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens comprises:
 combining the first set of encoded tokens and the second set of encoded tokens; and   generating, using a decoder of the third subnetwork, the enhanced current frame based on the combination of the first set of encoded tokens and the second set of encoded tokens.