Enhancing the resolution of a video stream

Abstract

In one embodiment, a method includes accessing first-resolution images corresponding to frames of a video, computing a motion vector based on a first-resolution image of a first frame in the video and a first-resolution image of a second frame in the video, generating a second-resolution warped image associated with the second frame by using the motion vector to warp a second-resolution reconstructed image associated with the first frame, generating a second-resolution intermediate image associated with the second frame based on the first-resolution image associated with the second frame, computing adjustment parameters by processing the first-resolution image associated with the second frame and the second-resolution warped image associated with the second frame using a machine-learning model, and adjusting pixels of the second-resolution intermediate image associated with the second frame based on the adjustment parameters to reconstruct a second-resolution reconstructed image associated with the second frame.

What is claimed is: 
     
       1. A method comprising, by a computing device:
 accessing first-resolution images corresponding to frames of a video; 
 computing a motion vector based on a first-resolution image of a first frame in the video and a first-resolution image of a second frame in the video; 
 generating a second-resolution warped image associated with the second frame by using the motion vector to warp a second-resolution reconstructed image associated with the first frame; 
 up-sampling the first-resolution image of the second frame using an interpolation technique to generate an up-sampled second-resolution image associated with the second frame; 
 computing adjustment parameters by processing the first-resolution image associated with the second frame and the second-resolution warped image associated with the second frame using a machine-learning model; and 
 adjusting pixels of the up-sampled second-resolution image associated with the second frame based on the adjustment parameters to reconstruct a second-resolution reconstructed image associated with the second frame. 
 
     
     
       2. The method of  claim 1 , wherein differences between the second-resolution reconstructed image associated with the second frame and a second-resolution ground truth image associated with the second frame are measured during a training process of the machine-learning model. 
     
     
       3. The method of  claim 2 , wherein gradients of trainable variables of the machine-learning model are computed based on the measured differences between the second-resolution reconstructed image associated with the second frame and the second-resolution ground truth image associated with the second frame. 
     
     
       4. The method of  claim 3 , wherein the trainable variables are updated by a gradient-descent backpropagation procedure. 
     
     
       5. The method of  claim 1 , wherein the first frame locates before the second frame in a frame sequence of the video. 
     
     
       6. The method of  claim 1 , wherein differences between selected pixels in a second-resolution warped image associated with a third frame and the selected pixels in a second-resolution ground truth image associated with the third frame are measured during a training process of the machine-learning model. 
     
     
       7. The method of  claim 6 , wherein the second frame locates before the third frame in a frame sequence of the video. 
     
     
       8. The method of  claim 7 , wherein the selected pixels are identified as pixels with strong optical flow correspondence by comparing pixels in a warped second-resolution ground truth image associated with the second frame and a warped second-resolution ground truth image associated with the third frame. 
     
     
       9. The method of  claim 7 , wherein the second-resolution warped image associated with the third frame is generated by:
 computing a second motion vector based on the second-resolution ground truth image associated with the second frame and the second-resolution ground truth image associated with the third frame; and 
 generating the second-resolution warped image associated with the third frame by using the second motion vector to warp the second-resolution reconstructed image associated with the second frame. 
 
     
     
       10. The method of  claim 6 , wherein gradients of trainable variables of the machine-learning model are computed based on the measured differences between the selected pixels in the second-resolution warped image associated with the third frame and the selected pixels in the second-resolution ground truth image associated with the third frame. 
     
     
       11. The method of  claim 10 , wherein the trainable variables are updated by a gradient-descent backpropagation procedure. 
     
     
       12. The method of  claim 1 , wherein a second resolution is higher than a first resolution. 
     
     
       13. The method of  claim 12 , wherein generating the up-sampled second-resolution image associated with the second frame comprises:
 uniformly placing pixels of the first-resolution image of the second frame into a second-resolution image plane for the up-sampled second-resolution image such that a plurality of pixels left blank; and 
 filling the plurality of blank pixels in the second-resolution image plane with interpolated values of non-blank neighboring pixels. 
 
     
     
       14. The method of  claim 1 , wherein the second-resolution warped image associated with the second frame comprises objects located at predicted locations based on the computed motion vector. 
     
     
       15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
 access first-resolution images corresponding to frames of a video; 
 compute a motion vector based on a first-resolution image of a first frame in the video and a first-resolution image of a second frame in the video; 
 generate a second-resolution warped image associated with the second frame by using the motion vector to warp a second-resolution reconstructed image associated with the first frame; 
 up-sample the first-resolution image of the second frame using an interpolation technique to generate an up-sampled second-resolution image associated with the second frame; 
 compute adjustment parameters by processing the first-resolution image associated with the second frame and the second-resolution warped image associated with the second frame using a machine-learning model; and 
 adjust pixels of the up-sampled second-resolution image associated with the second frame based on the adjustment parameters to reconstruct a second-resolution reconstructed image associated with the second frame. 
 
     
     
       16. The media of  claim 15 , wherein differences between the second-resolution reconstructed image associated with the second frame and a second-resolution ground truth image associated with the second frame are measured during a training process of the machine-learning model. 
     
     
       17. The media of  claim 16 , wherein gradients of trainable variables of the machine-learning model are computed based on the measured differences between the second-resolution reconstructed image associated with the second frame and the second-resolution ground truth image associated with the second frame. 
     
     
       18. The media of  claim 17 , wherein the trainable variables are updated by a gradient-descent backpropagation procedure. 
     
     
       19. The media of  claim 15 , wherein the first frame locates before the second frame in a frame sequence of the video. 
     
     
       20. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
 access first-resolution images corresponding to frames of a video; 
 compute a motion vector based on a first-resolution image of a first frame in the video and a first-resolution image of a second frame in the video; 
 generate a second-resolution warped image associated with the second frame by using the motion vector to warp a second-resolution reconstructed image associated with the first frame; 
 up-sample the first-resolution image of the second frame using an interpolation technique to generate an up-sampled second-resolution image associated with the second frame; 
 compute adjustment parameters by processing the first-resolution image associated with the second frame and the second-resolution warped image associated with the second frame using a machine-learning model; and 
 adjust pixels of the up-sampled second-resolution image associated with the second frame based on the adjustment parameters to reconstruct a second-resolution reconstructed image associated with the second frame.