Machine-learning based technique for fast image enhancement

Abstract

Systems and methods described herein may relate to image transformation utilizing a plurality of deep neural networks. An example method includes receiving, at a mobile device, a plurality of image processing parameters. The method also includes causing an image sensor of the mobile device to capture an initial image and receiving, at a coefficient prediction neural network at the mobile device, an input image based on the initial image. The method further includes determining, using the coefficient prediction neural network, an image transformation model based on the input image and at least a portion of the plurality of image processing parameters. The method additionally includes receiving, at a rendering neural network at the mobile device, the initial image and the image transformation model. Yet further, the method includes generating, by the rendering neural network, a rendered image based on the initial image, according to the image transformation model.

What is claimed is: 
     
       1. A method comprising:
 receiving, at a mobile device, a plurality of image processing parameters; 
 causing an image sensor of the mobile device to capture an initial image; 
 downsampling the initial image to provide an input image, wherein the input image comprises a downsampled version of the initial image; 
 subsequent to the downsampling, receiving, at a coefficient prediction neural network at the mobile device, the input image; 
 determining, using the coefficient prediction neural network, an image transformation model based on the input image and at least a portion of the plurality of image processing parameters; 
 receiving, at a rendering neural network at the mobile device, the initial image and the image transformation model; 
 generating, by the rendering neural network, a rendered image based on the initial image, according to the image transformation model; and 
 displaying the rendered image on a viewfinder of the mobile device. 
 
     
     
       2. The method of  claim 1 , wherein the initial image is a high-resolution image, wherein the input image is a low-resolution image, and wherein the input image comprises no more than 256 pixels along a first image dimension and no more than 256 pixels along a second image dimension. 
     
     
       3. The method of  claim 1 , wherein the image transformation model comprises a transformation data set, wherein the transformation data set includes at least five dimensions. 
     
     
       4. The method of  claim 3 , wherein the transformation data set includes a bilateral grid of affine matrices. 
     
     
       5. The method of  claim 3 , wherein the transformation data set has a form of a 16×16×8×3×4 data set. 
     
     
       6. The method of  claim 1 , wherein generating the rendered image is performed, at least in part, by an application programming interface running on at least one graphics processing unit. 
     
     
       7. The method of  claim 1 , wherein at least one of the coefficient prediction neural network or the rendering neural network is operable to carry out a Tensorflow inference runtime program. 
     
     
       8. A method comprising:
 during a training phase: 
 receiving at a server, a plurality of image pairs, wherein a first image of each image pair comprises a respective initial training image and wherein a second image of each image pair comprises a respective output training image; 
 determining, by the server, a plurality of image processing parameters based on the plurality of image pairs; and 
 transmitting the plurality of image processing parameters to a mobile device; and 
 during a prediction phase: 
 causing an image sensor of the mobile device to capture an initial image; 
 downsampling the initial image to provide an input image, wherein the input image comprises a downsampled version of the initial image; 
 subsequent to the downsampling, receiving, at a coefficient prediction neural network at the mobile device, the input image; 
 determining, using the coefficient prediction neural network, an image transformation model based on the input image and at least a portion of the plurality of image processing parameters; 
 receiving, at a rendering neural network at the mobile device, the initial image and the image transformation model; 
 generating, by the rendering neural network, a rendered image based on the initial image, according to the image transformation model; and 
 displaying the rendered image on a viewfinder of the mobile device. 
 
     
     
       9. The method of  claim 8 , wherein the initial image is a high-resolution image, wherein the input image is a low-resolution image, and wherein the input image comprises no more than 256 pixels along a first image dimension and no more than 256 pixels along a second image dimension. 
     
     
       10. The method of  claim 8 , wherein the image transformation model comprises a transformation data set, wherein the transformation data set includes at least five dimensions. 
     
     
       11. The method of  claim 10 , wherein the transformation data set includes a bilateral grid of affine matrices. 
     
     
       12. The method of  claim 10 , wherein the transformation data set has a form of a 16×16×8×3×4 data set. 
     
     
       13. The method of  claim 8 , wherein generating the rendered image is performed, at least in part, by an application programming interface running on at least one graphics processing unit, wherein the application programming interface comprises Open GL for Embedded Systems. 
     
     
       14. A mobile device comprising:
 an image sensor; 
 a viewfinder; 
 at least one tensor processing unit operable to execute instructions to carry out operations, the operations comprising: 
 receiving a plurality of image processing parameters; 
 causing the image sensor to capture an initial image; 
 downsampling the initial image to provide an input image, wherein the input image comprises a downsampled version of the initial image; 
 subsequent to the downsampling, receiving, at a coefficient prediction neural network, the input image; 
 determining, using the coefficient prediction neural network, an image transformation model based on the input image and at least a portion of the plurality of image processing parameters; 
 receiving, at a rendering neural network, the initial image and the image transformation model; 
 generating, by the rendering neural network, a rendered image based on the initial image, according to the image transformation model; and 
 displaying the rendered image on the viewfinder. 
 
     
     
       15. The mobile device of  claim 14 , wherein the tensor processing unit comprises at least one application-specific integrated circuit. 
     
     
       16. The mobile device of  claim 14 , wherein the tensor processing unit comprises at least one virtual machine having at least one persistent disk or memory resource. 
     
     
       17. The method of  claim 3 , wherein the transformation data set includes a three-dimensional grid of proper (3×4) matrices. 
     
     
       18. The method of  claim 10 , wherein the transformation data set includes a three-dimensional grid of proper (3×4) matrices.