Mobile device image item replacements

Abstract

A system for replacing physical items in images is discussed. A depicted item can be selected and removed from an image via image mask data and pixel merging techniques. Virtual light source positions can be generated based on real-world light source data from the image. A rendered simulation of a virtual item can then be integrated into the image to create a modified image for display.

1 .- 20 . (canceled) 
     
     
         21 . A method comprising:
 generating, using one or more processors of a user device, an image of a physical environment;   receiving, on a display device of the user device, a selection of an object to be replaced in the image;   determining a three-dimensional orientation of the object as depicted within the image using a pose detection neural network comprising a convolutional neural network trained to detect three-dimensional orientation of objects in a plurality of object training images, the objects of the plurality of object training images being of a same type as the object detected in the image;   removing, from the image, the object using regions that are proximate to the object in the image; and   generating a modified image that depicts a render of a virtual model that replaces the object in the physical environment.   
     
     
         22 . The method of  claim 21 , further comprising:
 generating the render of the virtual model in the three-dimensional orientation and as illuminated by one or more virtual light sources based on a lighting scheme in the image.   
     
     
         23 . The method of  claim 22 , further comprising:
 determining the lighting scheme of the image.   
     
     
         24 . The method of  claim 23 , wherein determining the lighting scheme comprises determining one or more bright regions of the image. 
     
     
         25 . The method of  claim 24 , further comprising:
 positioning, in a virtual environment, the one or more virtual light sources based on locations of the one or more bright regions of the image.   
     
     
         26 . The method of  claim 24 , wherein the determining of the one or more bright regions of the image comprises determining an area of pixels in the image having higher brightness values. 
     
     
         27 . The method of  claim 21 , wherein, in the image, the object is depicted in an object image region, and the regions that are proximate to the object in the image are proximate regions that are external to the object image region. 
     
     
         28 . The method of  claim 27 , wherein the object is removed by merging the proximate regions and the object image region. 
     
     
         29 . The method of  claim 28 , wherein the proximate regions and the object image region are merged using a neural network that implements partial convolutional layers. 
     
     
         30 . The method of  claim 27 , wherein the object is removed by interpolating the proximate regions and the object image region. 
     
     
         31 . The method of  claim 21 , further comprising:
 displaying the image on a display device of the user device; and   receiving selection of the object through the display device of the user device.   
     
     
         32 . The method of  claim 31 , wherein receiving selection of the object comprises receiving selection of a selected region of the image that depicts the object. 
     
     
         33 . The method of  claim 32 , further comprising:
 generating an image mask using the selected region.   
     
     
         34 . The method of  claim 32 , further comprising:
 segmenting the image into segment regions using an image segmentation convolutional neural network (CNN), wherein the selected region is identified from a user input on the image as displayed on the display device of the user device.   
     
     
         35 . The method of  claim 34 , wherein the user input is one of: a tap gesture or a click. 
     
     
         36 . The method of  claim 32 , wherein receiving selection of the object through the display device comprises:
 receiving, on the display device of the user device, a swipe gesture over at least a portion of the object as depicted in the image.   
     
     
         37 . A system comprising:
 one or more processors;   a display device   a memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:   generating an image of a physical environment;   receiving, on the display device, a selection of an object to be replaced in the image;   determining a three-dimensional orientation of the object as depicted within the image using a pose detection neural network comprising a convolutional neural network trained to detect three-dimensional orientation of objects in a plurality of object training images, the objects of the plurality of object training images being of a same type as the object detected in the image;   removing, from the image, the object using regions that are proximate to the object in the image; and   generating a modified image that depicts a render of a virtual model that replaces the object in the physical environment.   
     
     
         38 . The system of  claim 37 , the operations further comprising:
 generating the render of the virtual model in the three-dimensional orientation and as illuminated by one or more virtual light sources based on a lighting scheme in the image.   
     
     
         39 . The system of  claim 38 , the operations further comprising:
 determining the lighting scheme of the image.   
     
     
         40 . A machine-readable storage device embodying instructions that, when executed by a device, cause the device to perform operations comprising:
 generating an image of a physical environment;   receiving, on a display device, a selection of an object to be replaced in the image;   determining a three-dimensional orientation of the object as depicted within the image using a pose detection neural network comprising a convolutional neural network trained to detect three-dimensional orientation of objects in a plurality of object training images, the objects of the plurality of object training images being of a same type as the object detected in the image;   removing, from the image, the object using regions that are proximate to the object in the image; and   generating a modified image that depicts a render of a virtual model that replaces the object in the physical environment.