Generating gaze corrected images using bidirectionally trained network

Abstract

An example apparatus for adjusting eye gaze in images one or more processors to execute instructions to bidirectionally train a neural network; access a target angle and an input image, the input image including an eye in a first position; generate a vector field with the neural network; and generate a gaze-adjusted image based on the vector field, the gaze-adjusted image including the eye in a second position.

1 . (canceled) 
     
     
         2 . An apparatus comprising:
 memory;   instructions; and   programmable circuitry to execute the instructions to:
 generate a vector field using a first neural network based on a target angle and an input image, the input image including an eye in a first position; 
 generate a gaze-adjusted image based on the vector field, the gaze-adjusted image including the eye in a second position; and 
 enhance photorealism of the gaze-adjusted image using a second neural network. 
   
     
     
         3 . The apparatus of  claim 2 , wherein the second neural network is at least a part of a generative adversarial network. 
     
     
         4 . The apparatus of  claim 2 , wherein the second neural network is to learn a mapping of synthetic images and the input image. 
     
     
         5 . The apparatus of  claim 4 , wherein the second neural network generates pixel-perfect labels for the synthetic images. 
     
     
         6 . The apparatus of  claim 2 , wherein the second neural network is trained on based on analysis of a labeled data set by a third neural network. 
     
     
         7 . The apparatus of  claim 2 , wherein the second neural network is trained on a data set of wholly synthetic images. 
     
     
         8 . The apparatus of  claim 2 , wherein the second neural network is to use mean absolute error losses defined between the inputs and outputs of the second neural network. 
     
     
         9 . The apparatus of  claim 8 , wherein the second neural network is to use the mean absolute error losses as reconstruction losses. 
     
     
         10 . The apparatus of  claim 2 , wherein the second neural network is to modify color while preserving gaze direction. 
     
     
         11 . At least one computer readable storage device comprising instructions that, when executed, cause one or more processors to at least:
 generate a vector field using a first neural network based on a target angle and an input image, the input image including an eye in a first position;   generate a gaze-adjusted image based on the vector field, the gaze-adjusted image including the eye in a second position; and   enhance photorealism of the gaze-adjusted image using a second neural network.   
     
     
         12 . The computer readable storage device of  claim 11 , wherein the second neural network is at least a part of a generative adversarial network. 
     
     
         13 . The computer readable storage device of  claim 11 , wherein the instructions cause the second neural network to learn a mapping of synthetic images and the input image. 
     
     
         14 . The computer readable storage device of  claim 13 , wherein the instructions cause the second neural network to generate pixel-perfect labels for the synthetic images. 
     
     
         15 . The computer readable storage device of  claim 11 , wherein the second neural network is trained on based on analysis of a labeled data set by a third neural network. 
     
     
         16 . The computer readable storage device of  claim 11 , wherein the instructions cause the second neural network to use mean absolute error losses defined between the inputs and outputs of the second neural network. 
     
     
         17 . The computer readable storage device of  claim 16 , wherein the instructions cause the second neural network to use the mean absolute error losses as reconstruction losses. 
     
     
         18 . The computer readable storage device of  claim 11 , wherein the instructions cause the second neural network to modify color while preserving gaze direction. 
     
     
         19 . A method to generate a gaze corrected image, the method comprising:
 generating a vector field with a first neural network based on a target angle and an input image, the input image including an eye in a first position;   generating a gaze-adjusted image based on the vector field, the gaze-adjusted image including the eye in a second position; and   enhancing photorealism of the gaze-adjusted image using a second neural network.   
     
     
         20 . The method of  claim 19 , further including training the second neural network with synthetic images. 
     
     
         21 . The method of  claim 20 , further including generating pixel-perfect labels for the synthetic images using the second neural network. 
     
     
         22 . The method of  claim 19 , further including modifying color while maintaining gaze.