Super resolution using a generative adversarial network

Abstract

A neural network is trained to process received visual data to estimate a high-resolution version of the visual data using a training dataset and reference dataset. A set of training data is generated, and a generator convolutional neural network parameterized by first weights and biases is trained by comparing characteristics of the training data to characteristics of the reference dataset. The first network is trained to generate super-resolved image data from low-resolution image data and the training includes modifying first weights and biases to optimize processed visual data based on the comparison between the characteristics of the training data and the characteristics of the reference dataset. A discriminator convolutional neural network parameterized by second weights and biases is trained by comparing characteristics of the generated super-resolved image data to characteristics of the reference dataset, and where the second network is trained to discriminate super-resolved image data from real image data.

What is claimed is: 
     
         1 . A method comprising:
 receiving an initial image;   generating a super-resolution image from the initial image by using a generator convolutional neural network trained to minimize perceptual loss from the initial image, the generator convolutional neural network being parameterized by first weights and biases selected to optimize processed visual data based on the comparison between the one or more characteristics of visual image training data and the one or more characteristics of a visual image reference dataset and by using a discriminator convolutional neural network that is parameterized by second weights and biases, wherein the discriminator convolutional neural network is trained to discriminate super-resolved image data from real image data; and   storing the generated super-resolution image.   
     
     
         2 . The method of  claim 1 , wherein using the generator convolutional neural network trained to minimize perceptual loss includes using a generator convolutional neural network that minimizes a Euclidean distance between feature representations of an image that is reconstructed from a downsampled version of a reference image and the reference image. 
     
     
         3 . The method of  claim 1 , wherein the generator convolutional neural network uses a perceptual loss function that is a weighted combination of content loss and adversarial loss. 
     
     
         4 . The method of  claim 1 , wherein the generator convolutional neural network uses a perceptual loss function that is a weighted combination of content loss, adversarial loss, and regularization loss. 
     
     
         5 . The method of  claim 1 , wherein using the generator convolutional neural network trained to minimize perceptual loss includes using a visual geometry group neural network. 
     
     
         6 . A computer-readable medium storing generator convolutional neural network and a discriminator convolutional neural network trained to generate an image using a method comprising:
 receiving an initial image;
 generating a super-resolution image from the initial image by using a generator convolutional neural network trained to minimize perceptual loss from the initial image, the generator convolutional neural network being parameterized by first weights and biases selected to optimize processed visual data based on the comparison between the one or more characteristics of visual image training data and the one or more characteristics of a visual image reference dataset and by using a discriminator convolutional neural network that is parameterized by second weights and biases, wherein the discriminator convolutional neural network is trained to discriminate super-resolved image data from real image data; and 
 storing the generated super-resolution image. 
   
     
     
         7 . The computer-readable medium of  claim 6 , wherein using the generator convolutional neural network trained to minimize perceptual loss includes using a generator convolutional neural network that minimizes a Euclidean distance between feature representations of an image that is reconstructed from a downsampled version of a reference image and the reference image. 
     
     
         8 . The computer-readable medium of  claim 6 , wherein the generator convolutional neural network uses a perceptual loss function that is a weighted combination of content loss and adversarial loss. 
     
     
         9 . The computer-readable medium of  claim 6 , wherein the generator convolutional neural network uses a perceptual loss function that is a weighted combination of content loss, adversarial loss, and regularization loss. 
     
     
         10 . The computer-readable medium of  claim 6 , wherein using the generator convolutional neural network trained to minimize perceptual loss includes using a visual geometry group neural network.