Real-time intelligent image manipulation system

Abstract

A system for manipulating images according to styles chosen by a user includes a feed-forward image manipulation model for everyday use and an optimization image manipulation model for more professional use. The optimization image manipulation model optimizes directly over output image pixels to minimize both the content loss and style loss. Users can select their own content and style images, and can choose between using the feed-forward image manipulation model and optimization image manipulation model.

What is claimed is: 
     
         1 . A system for manipulating images according to user-chosen styles, comprising:
 a memory;   an interface device in communication with the memory and configured to allow a user to:
 upload one or more images to the memory, 
 select a content image from the one or more images, 
 select a style image from the one or more images, and 
 set one or more process parameters stored in the memory; 
   an encoder network comprising multiple neural computing layers configured to generate a content feature vector by extracting content features from the content image according to the process parameters and generate a style feature vector by extracting style features from the style image according to the process parameters;   a transformation module configured to generate a stylized feature vector based on the content feature vector and the style feature vector; and   a decoder module configured to generate an output image based on the stylized feature vector;   wherein the interface is further configured to display the output image to the user.   
     
     
         2 . The system of  claim 1 , wherein:
 the process parameters include an indication for the system to use an optimization-based model;   the decoder module is further configured to generate a tentative output image based on the stylized feature vector;   the encoder network is further configured to:
 receive the tentative output image from the decoder, 
 generate a tentative content feature vector by extracting content features from the tentative output image, 
 generate a tentative style feature vector by extracting style features from the tentative output image; 
   the transformation module is further configured to:
 generate a refined feature vector based on the content feature vector, tentative content feature vector, style feature vector, and tentative style feature vector; and 
   the decoder module is further configured to:
 generate a refined output image based on the refined feature vector, and 
 until an optimization parameter is reached, repeatedly perform an optimization iteration by sending the refined output image as the tentative output image to the encoder network and receiving the resulting refined feature vector from the transformation module. 
   
     
     
         3 . The system of  claim 2 , wherein the decoder module is further configured to repeatedly send the refined output image to the interface to be displayed with each optimization iteration. 
     
     
         4 . The system of  claim 3 , wherein:
 the optimization parameter is configured to be set by the user;   the interface is configured to, after displaying the refined output image, prompt the user for a choice whether to advance optimization or stop optimization, thereby setting the optimization parameter.   
     
     
         5 . The system of  claim 2 , wherein the transformation model is further configured to:
 compare the content feature vector to the tentative content feature vector to determine a content loss parameter;   compare the style feature vector to the tentative style feature vector to determine a style loss parameter; and   generate the refined feature vector by adjusting the tentative content feature vector based on the content loss parameter and adjusting the tentative style feature vector based on the style loss parameter.   
     
     
         6 . The system of  claim 1 , wherein:
 the process parameters include an indication for the system to use a feed-forward model;   generating the content feature vector by extracting content features from the content image includes generating an initial content image vector and performing one or more whitening transformations on the initial content image vector; and   generating the stylized feature vector by the transformation module includes performing a coloring transformation on the content feature vector.   
     
     
         7 . The system of  claim 6 , wherein each of the neural computing layers of the encoder network is configured to perform one of the one or more whitening transformations, with each respective whitening transformation being performed at a different granularity level. 
     
     
         8 . The system of  claim 1 , further comprising a distribution module configured to distribute operations of at least one of the encoder network, transformation module, or decoder module to an external processor. 
     
     
         9 . The system of  claim 1 , wherein:
 one of the process parameters set by the user includes a content amount parameter indicative of the user's desired amount of content features in the output image, and a style amount parameter indicative of the user's desired amount of style features in the output image; and   the transformation module is configured to generate a stylized feature vector further based on content amount parameter and style amount parameter.   
     
     
         10 . The system of  claim 1 , wherein the interface is configured to allow the user to select a plurality of content images from the one or more images, and the interface is configured to display a resulting plurality of output images corresponding to the plurality of content images. 
     
     
         11 . A method of image manipulation based on a user's desired content and style, comprising:
 obtaining a content image representative of the user's desired content;   obtaining a style image representative of the user's desired style;   obtaining a choice of either a quick processing model or a professional processing model;   generating a stylized feature vector;   decoding the stylized feature vector into an output image having the user's desired content and style; and   displaying the output image;   wherein, if the quick processing model is chosen, generating the stylized feature vector comprises:
 encoding the content image into a content feature vector; 
 encoding the style image into a style feature vector; and 
 generating the stylized feature vector based on the content feature vector and the style feature vector; 
   and wherein, if the professional processing model is chosen, generating the stylized feature vector comprises:
 determining an optimization parameter; 
 encoding the content image into a tentative feature vector; 
 encoding the style image into a tentative style feature vector, 
 generating a tentative stylized feature vector based on the tentative content feature vector and tentative style feature vector, 
 applying the tentative stylized feature vector to generate a tentative output image; 
 repeating an optimization operation for a number of optimization iterations determined by the optimization parameter, the optimization operation comprising:
 encoding content elements of the tentative output image into a refined content feature vector, 
 encoding style elements of the tentative output image into a refined style feature vector, 
 determining a content loss parameter by comparing the tentative content feature vector to the refined content feature vector, 
 determining a style loss parameter by comparing the tentative style feature vector to the refined style feature vector, 
 optimizing the refined content feature vector and refined style feature vector based on the content loss parameter and style loss parameter, 
 generating a refined stylized feature vector based on the refined content feature vector and refined style feature vector, 
 if the refined stylized feature vector is not sufficiently optimized according to the optimization parameter, decoding the refined stylized feature vector into a refined output image, saving the refined output image as the tentative output image, and repeating the optimization operation, and 
 if the refined stylized feature vector is sufficiently optimized according to the optimization parameter, saving the refined stylized feature vector as the stylized feature vector. 
 
   
     
     
         12 . The method of  claim 11  wherein, if the professional processing model is chosen, the optimization parameter is based on a desired level of optimization chosen by the user. 
     
     
         13 . The method of  claim 11  wherein, if the professional processing model is chosen, the optimization operation further comprises displaying the refined output image with each optimization iteration. 
     
     
         14 . The method of  claim 11  wherein, if the quick processing model is chosen, the content feature vector has a corresponding covariance matrix which is an identity matrix, and the stylized feature vector has the same covariance matrix. 
     
     
         15 . The method of  claim 11  wherein, if the quick processing model is chosen, encoding the content image into the content feature vector includes performing a feed-forward pass to remove style elements present in the content image. 
     
     
         16 . The method of  claim 15 , wherein the feed-forward pass to remove style elements present in the content image includes performing a whitening transformation on the content feature vector, and generating the stylized feature vector based on the content feature vector and the style feature vector includes performing a coloring transformation on the content feature vector based on the style feature vector. 
     
     
         17 . The method of  claim 16 , wherein the whitening transformation and coloring transformation are each performed a plurality of times on the content image at different levels of granularity. 
     
     
         18 . The method of  claim 11 , further comprising:
 obtaining a number of content images;   generating a plurality of stylized feature vectors including a respective stylized feature vector for each of the number of content images;   applying the plurality of respective stylized feature vectors to generate respective output images; and   displaying the output images, each output image having a respective display position and display size chosen based on the number of content images such that the output images are arranged in a grid.   
     
     
         19 . The method of  claim 11 , wherein obtaining the style image includes the user uploading the style image. 
     
     
         20 . A non-transitory computer-readable medium including instructions, which when executed by one or more processors, cause the processors to perform a method, comprising:
 obtaining a content image representative of a user's desired content;   obtaining a style image representative of the user's desired style;   obtaining a choice of either a quick processing model or a professional processing model;   generating a stylized feature vector;   applying the stylized feature vector to generate an output image having the user's desired content and style; and   displaying the output image;   wherein, if the quick processing model is chosen, generating the stylized feature vector comprises:
 encoding the content image into a content feature vector; 
 encoding the style image into a style feature vector; and 
 generating the stylized feature vector based on the content feature vector and the style feature vector; 
   and wherein, if the professional processing model is chosen, generating the stylized feature vector comprises:
 determining an optimization parameter; 
 encoding the content image into a tentative feature vector; 
 encoding the style image into a tentative style feature vector, 
 generating a tentative stylized feature vector based on the tentative content feature vector and tentative style feature vector, 
 applying the tentative stylized feature vector to generate a tentative output image; 
 repeating an optimization operation for a number of optimization iterations determined by the optimization parameter, the optimization operation comprising:
 encoding content elements of the tentative output image into a refined content feature vector, 
 encoding style elements of the tentative output image into a refined style feature vector, 
 determining a content loss parameter by comparing the tentative content feature vector to the refined content feature vector, 
 determining a style loss parameter by comparing the tentative style feature vector to the refined style feature vector, 
 optimizing the refined content feature vector and refined style feature vector based on the content loss parameter and style loss parameter, 
 generating a refined stylized feature vector based on the refined content feature vector and refined style feature vector, 
 if the refined stylized feature vector is not sufficiently optimized according to the optimization parameter, applying the refined stylized feature vector to generate a refined output image, saving the refined output image as the tentative output image, and repeating the optimization operation, and 
 if the refined stylized feature vector is sufficiently optimized according to the optimization parameter, saving the refined stylized feature vector as the stylized feature vector.