System and Method for Generating Two-Dimensional (2D) Image of an Object in a Scene

Abstract

An AI image processing system is provided. The system is configured to receive latent parameters defining representations of attributes of an object in an image. The latent parameters include subsets of latent parameters, and each latent parameter subset corresponds to one attribute. The system is configured to produce a joint encoding of the latent parameters and produce an implicit 3D model of the object based on the joint encoding. The implicit 3D model defines a volumetric representation for each 3D location in the scene with respect to an input viewing direction. The volumetric representation defines a signed distance value and/or an optical density, and a color and/or a feature vector. The system is configured to render a 2D image of the scene including the object viewed from the input viewing direction based on the produced volumetric representations for different 3D locations in the scene.

What claimed is: 
     
         1 . An artificial intelligence (AI) image processing system for generating and rendering an image of a scene including an object viewed from one or multiple viewing directions, the AI image processing system comprising: at least one processor and a memory having instructions stored thereon that cause the at least one processor of the AI image processing system to:
 receive a set of latent parameters defining representations of at least two attributes of a shape, an albedo, an illumination, a pose, and a background of the object in the image, wherein the set of latent parameters includes multiple subsets of latent parameters, wherein each latent parameter subset corresponds to one of the at least two attributes of the object in the image;   submit the set of latent parameters into a renderer mapping neural network trained to produce a joint encoding of the set of latent parameters;   submit the joint encoding to a deep implicit function neural network trained to produce, given the joint encoding, an implicit three-dimensional (3D) model of the object defining a volumetric representation for each 3D location in the scene including the object with respect to an input viewing direction, wherein the volumetric representation defines one or a combination of a signed distance value and an optical density, and one or a combination of a color and a feature vector; and   submit the volumetric representations produced for different 3D locations in the scene to an image renderer configured to render a 2D image of the scene including the object viewed from the input viewing direction.   
     
     
         2 . The AI image processing system of  claim 1 , wherein, to modify an attribute of the object in the 2D image of the scene, the processor is configured to:
 modify a latent parameter from the set of latent parameters corresponding to the attribute of the object; and   generate a modified image of the object based on a combination of the modified latent parameter with the remaining unmodified latent parameters from the set of latent parameters.   
     
     
         3 . The AI image processing system of  claim 2 , wherein, to modify the attribute of the object, the processor is configured to:
 sample a probabilistic distribution of the latent parameter corresponding to the attribute of the object to produce the modified latent parameter.   
     
     
         4 . The AI image processing system of  claim 2 , wherein the memory stores a mapping between values of the set of latent parameters and corresponding values of the attribute of the object, and wherein, to modify the attribute of the object, the processor is configured to:
 receive a desired value of the attribute to be modified in the image of the object; and   determine a desired value of the modified latent parameter corresponding to the desired value of the attribute using the stored mapping.   
     
     
         5 . The AI image processing system of  claim 4 , wherein the mapping is based on a statistical analysis of the distribution of images in a dataset. 
     
     
         6 . The AI image processing system of  claim 5 , wherein the mapping is derived using principal components analysis (PCA) of values of the attribute in a dataset of training images of the object. 
     
     
         7 . The AI image processing system of  claim 2 , wherein the processor is configured to:
 obtain the modified latent parameter to match parameters of an input image by using an object encoder, wherein the object encoder is configured to determine a numerical value of the corresponding attribute in the input image.   
     
     
         8 . The AI image processing system of  claim 2 , wherein the processor is configured to:
 train the deep implicit function neural network using an object encoder that determines numerical values of attributes from a training image of a training object.   
     
     
         9 . The AI image processing system of  claim 8 , wherein the processor is configured to:
 train the deep implicit function neural network using the object encoder based on a set of parametric consistency loss functions to enforce consistency between the received set of latent parameters for generating the image of a scene and parameters that encode attributes of the rendered 2D image of the scene.   
     
     
         10 . The AI image processing system of  claim 8 , wherein the processor is configured to:
 train the deep implicit function neural network using the object encoder based on a set of image-based consistency loss functions to minimize a correlation between a latent parameter relating to the background of the training object and an appearance of the training object, and to minimize a correlation between a latent parameter relating to the appearance of the training object and the appearance of the background of the training object.   
     
     
         11 . The AI image processing system of  claim 8 , wherein the object is a human head. 
     
     
         12 . The AI image processing system of  claim 11 , wherein the processor is configured to:
 train the deep implicit function neural network using the object encoder based on a set of image-based consistency loss functions to minimize a correlation between a latent parameter or the set of latent parameters relating to hair and background of the human head and an appearance of a face, and to minimize a correlation between a latent parameter or the set of latent parameters relating other attributes of the human head and the appearance of the hair and background of the human head.   
     
     
         13 . The AI image processing system of  claim 11 , wherein the processor is configured to:
 partition a multi-dimensional latent code of the scene into the set of latent parameters defining representations of at least two attributes of the object in the image; and   introduce an additional latent parameter as part of the set of latent parameters, the additional latent parameter defining variable representation of a facial appearance factor of the human head.   
     
     
         14 . The AI image processing system of  claim 1 , wherein the processor is configured to execute modules of the AI image processing system comprising:
 the renderer mapping neural network to generate the joint encoding based on the set of latent parameters defining representations of the at least two attributes of the object in the image;   the deep implicit neural network and image renderer configured to generate one or more of one or more low-resolution images, a low-resolution feature map and one or more low-resolution signed distance function (SDF) surfaces, based on the input viewing direction and the joint encoding of the set of latent parameters;   the generator mapping neural network configured to convert the joint representation into a style code that modulates a high-resolution image generator; and   the high-resolution image generator configured to render the high-resolution 2D image of the scene including the object viewed from the input viewing direction.   
     
     
         15 . The AI image processing system of  claim 14 , wherein the high-resolution image generator is configured to render a high-resolution 2D image of the scene including the object viewed from the input viewing direction, instead of or in addition to the 2D image of the scene rendered by the image renderer. 
     
     
         16 . The AI image processing system of  claim 1 , wherein the shape attribute represents a 3-dimensional (3D) shape of the object, and the pose attribute represents a pose of the object in 3D space. 
     
     
         17 . The AI image processing system of  claim 1 , wherein the lighting attribute utilizes a spherical harmonics representation. 
     
     
         18 . The AI image processing system of  claim 1 , wherein the deep implicit function neural network is one of: a signed distance function (SDF) neural network, a neural radiance field (NeRF) neural network, or a Gaussian splatting neural network. 
     
     
         19 . A method for generating and rendering an image of a scene including an object viewed from one or multiple viewing directions, the method comprising:
 receiving a set of latent parameters defining representations of at least two attributes of a shape, an albedo, an illumination, a pose, and a background of the object in the image, wherein the set of latent parameters includes multiple subsets of latent parameters, wherein each latent parameter subset corresponds to one of the at least two attributes of the object in the image;   submitting the set of latent parameters into a renderer mapping neural network trained to produce a joint encoding of the set of latent parameters;   submitting the joint encoding to a deep implicit function neural network trained to produce, given the joint encoding, an implicit three-dimensional (3D) model of the object defining a volumetric representation for each 3D location in the scene including the object with respect to an input viewing direction, wherein the volumetric representation defines one or a combination of a signed distance value and an optical density, and one or a combination of a color and a feature vector; and   submitting the volumetric representations produced for different 3D locations in the scene to an image renderer configured to render on a display device a 2D image of the scene including the object viewed from the input viewing direction.   
     
     
         20 . A non-transitory computer readable storage medium embodied thereon a program executable by a processor for performing a method, the method comprising:
 receiving a set of latent parameters defining representations of at least two attributes of a shape, an albedo, an illumination, a pose, and a background of the object in the image, wherein the set of latent parameters includes multiple subsets of latent parameters, wherein each latent parameter subset corresponds to one of the at least two attributes of the object in the image;   submitting the set of latent parameters into a renderer mapping neural network trained to produce a joint encoding of the set of latent parameters;   submitting the joint encoding to a deep implicit function neural network trained to produce, given the joint encoding, an implicit three-dimensional (3D) model of the object defining a volumetric representation for each 3D location in the scene including the object with respect to an input viewing direction, wherein the volumetric representation defines one or a combination of a signed distance value and an optical density, and one or a combination of a color and a feature vector; and   submitting the volumetric representations produced for different 3D locations in the scene to an image renderer configured to render on a display device a 2D image of the scene including the object viewed from the input viewing direction.