Method and apparatus for applying deep learning techniques in video coding, restoration and video quality analysis (vqa)

Abstract

Video quality analysis may be used in many multimedia transmission and communication applications, such as encoder optimization, stream selection, and/or video reconstruction. An objective VQA metric that accurately reflects the quality of processed video relative to a source unprocessed video may take into account both spatial measures and temporal, motion-based measures when evaluating the processed video. Temporal measures may include differential motion metrics indicating a difference between a frame difference of a plurality of frames of the processed video relative to that of a corresponding plurality of frames of the source video. In addition, neural networks and deep learning techniques can be used to develop additional improved VQA metrics that take into account both spatial and temporal aspects of the processed and unprocessed videos.

What is claimed is: 
     
         1 . A computer-implemented method for selecting a processed video, comprising:
 receiving, by a streaming server comprising one or more processors and memory, an unprocessed video comprising a first plurality of frames;   generating a plurality of processed videos from the unprocessed video using a plurality of encoding schemes, each processed video generated by applying a respective encoding scheme of the plurality of encoding schemes to the unprocessed video;   generating, for each of the plurality of processed videos, an aggregate quality or loss measure indicating a level of quality of the processed video relative to the unprocessed video, by:
 decoding or restoring the processed video to create a reconstructed video; 
 identifying a second plurality of frames of the reconstructed video corresponding to the first plurality of frames of the unprocessed video; 
 generating at least one spatial loss measure indicating a level of difference between each of at least a portion of the first plurality of frames and respective corresponding frames of the second plurality of frames; 
 determining one or more first motions associated with the unprocessed video, each first motion based upon one or more frames of the first plurality of frames; 
 determining one or more second motions associated with the reconstructed video, each second motion corresponding to a first motion of the one or more first motions, and based upon one or more corresponding frames of the second plurality of frames; 
 generating at least one temporal loss measure indicating a level of difference between the one or more first motions associated with the unprocessed video and the one or more second motions associated with the reconstructed video; and 
 combining the generated at least one spatial measure and at least one temporal measure to determine the aggregate quality or loss measure for the reconstructed video; and 
   selecting a processed video of the plurality of processed videos, based upon the aggregate quality or loss measures determined for each of the reconstructed videos;   performing one or more of transmitting the selected processed video or storing the selected processed video on a storage device.   
     
     
         2 . The method of  claim 1 , wherein a first motion of the one or more first motions indicates a first frame difference between at least two frames of the first plurality of frames, and a second motion of the one or more second motions indicates a second frame difference between corresponding at least two frames of the second plurality of frames. 
     
     
         3 . The method of  claim 2 , wherein the first and second frame differences are computed at the frame, slice, tile, or block level. 
     
     
         4 . The method of  claim 2 , wherein a level of difference between the one or more first motions and the one or more second motions is an absolute difference between the first frame difference and the second frame difference. 
     
     
         5 . The method of  claim 1 , wherein the aggregate quality or loss measure corresponds to a linear combination of the at least one spatial measure and at least one temporal measure, the linear combination having coefficients determined using a support vector machine. 
     
     
         6 . The method of  claim 1 , wherein the aggregate quality or loss measure corresponds to a nonlinear combination of the at least one spatial measure and at least one temporal measure, the nonlinear combination determined using a neural network. 
     
     
         7 . The method of  claim 6 , further comprising:
 training the neural network configured to receive at least one spatial measure and at least one temporal measure and determining weights and biases for combining the at least one spatial measure and at least one temporal measure for generating an aggregate quality measure, based upon a training set of qualitative quality measures each indicating a human perceived level of quality of a processed video relative to a corresponding unprocessed video.   
     
     
         8 . The method of  claim 1 , wherein combining the generated at least one spatial measure and at least one temporal measure comprises combining the at least one spatial measure and at least one temporal measure using a non-linear compound function. 
     
     
         9 . The method of  claim 8 , wherein the non-linear compound function corresponds to a cascade of a first linear function followed by a second nonlinear function. 
     
     
         10 . The method of  claim 1 , wherein combining the generated at least one spatial measure and at least one temporal measure comprises inputting the at least one spatial measure and at least one temporal measure into a trained neural network model configured to determine a function for combining the at least one spatial measure and at least one temporal measure to generate an aggregate quality measure, wherein the neural network model is trained based upon a training set of qualitative quality measures indicating a level of human-perceived quality of processed videos relative to corresponding unprocessed videos. 
     
     
         11 . The method of  claim 1  wherein the processing of videos pertains to encoding of videos. 
     
     
         12 . A method for optimizing an encoder for encoding video, comprising:
 receiving an unencoded video;   encoding the unencoded video at an encoder using a first encoding scheme to generate an encoded video, wherein one or more parameters of the first encoding scheme are selected based upon a distortion measure indicating a level of distortion of the encoded video relative to the unencoded video;   wherein the distortion measure corresponds to a combination of one or more spatial measures indicating a level of difference between a plurality of corresponding frames of the encoded video and the unencoded video, and one or more temporal measures indicating a level of difference between a first frame difference between a plurality of frames of the unencoded video and a second frame difference between a corresponding plurality of frames of the encoded video, and   wherein the spatial measures and the temporal measures are combined using a machine learning method.   
     
     
         13 . The method of  claim 12 , wherein the machine learning method is a support vector machine, which determines coefficients for a linear combination of the one or more spatial and the one or more temporal measures. 
     
     
         14 . The method of  claim 12 , wherein the machine learning method is a trained neural network, which determines a nonlinear functional combination of the one or more spatial and the one or more temporal measures. 
     
     
         15 . A computer-implemented method for restoring a processed video, comprising:
 receiving, by a video restoration computer comprising one or more processors and memory, a processed video comprising a first plurality of frames;   generating a plurality of restored videos from the processed video using a plurality of restoration schemes, each restored video generated by
 decoding as appropriate the processed video to create a reconstructed video; 
 and applying a respective restoration scheme of the plurality of restoration schemes to the reconstructed video; 
   generating, for each of the plurality of restored videos, an aggregate quality or loss measure indicating a level of quality of the restored video relative to the unprocessed video, by:
 identifying a second plurality of frames of the restored video corresponding to the first plurality of frames of the unprocessed video; 
 generating at least one spatial quality or loss measure indicating a level of difference between each of at least a portion of the first plurality of frames and respective corresponding frames of the second plurality of frames; 
 determining one or more first motions associated with the unprocessed video, each first motion based upon one or more frames of the first plurality of frames; 
 determining one or more second motions associated with the processed and reconstructed video, each second motion corresponding to a first motion of the one or more first motions, and based upon one or more corresponding frames of the second plurality of frames; 
 generating at least one temporal quality or loss measure indicating a level of difference between the one or more first motions associated with the unprocessed video and the one or more second motions associated with the restored video; and 
 combining the generated at least one spatial measure and at least one temporal measure to determine the aggregate quality or loss measure for the restored video; and 
   selecting a restored video of the plurality of restored videos, based upon the aggregate quality or loss measures determined for each of the restored videos;   performing one or more of displaying the selected restored video or storing the selected restored video on a storage device.   
     
     
         16 . The method of  claim 15 , wherein a first motion of the one or more first motions indicates a first frame difference between at least two frames of the first plurality of frames, and a second motion of the one or more second motions indicates a second frame difference between corresponding at least two frames of the second plurality of frames. 
     
     
         17 . The method of  claim 15 , wherein the aggregate quality or loss measure corresponds to a linear combination of the at least one spatial measure and at least one temporal measure, the linear combination having coefficients determined using a support vector machine. 
     
     
         18 . The method of  claim 15 , wherein the aggregate quality or loss measure corresponds to a nonlinear combination of the at least one spatial measure and at least one temporal measure, the nonlinear combination determined using a neural network. 
     
     
         19 . The method of  claim 15 , wherein combining the generated at least one spatial measure and at least one temporal measure comprises combining the at least one spatial measure and at least one temporal measure using a non-linear compound function. 
     
     
         20 . The method of  claim 15 , wherein combining the generated at least one spatial measure and at least one temporal measure comprises inputting the at least one spatial measure and at least one temporal measure into a trained neural network model configured to determine a function for combining the at least one spatial measure and at least one temporal measure to generate an aggregate quality measure, wherein the neural network model is trained based upon a training set of qualitative quality measures indicating a level of human-perceived quality of processed videos relative to corresponding unprocessed videos.