Dense feature scale detection for image matching

Abstract

Dense feature scale detection can be implemented using multiple convolutional neural networks trained on scale data to more accurately and efficiently match pixels between images. An input image can be used to generate multiple scaled images. The multiple scaled images are input into a feature net, which outputs feature data for the multiple scaled images. An attention net is used to generate an attention map from the input image. The attention map assigns emphasis as a soft distribution to different scales based on texture analysis. The feature data and the attention data can be combined through a multiplication process and then summed to generate dense features for comparison.

What is claimed is: 
     
       1. A method comprising:
 identifying, using one or more processors of a machine, an image; 
 generating a plurality of scaled images from the image; 
 generating image features for the plurality of scaled images; 
 generating a dense features by combining image features with attention values of an attention map, the attention values being one or more numerical values that modify values of the dense features based at least in part on the scale of the plurality of scaled images; 
 identifying, using the dense features, a location within one or more images of an object depicted in each of the one or more images; 
 storing, in memory, the location of the object within each of the one or more images; and 
 generating one or more modified images from the one or more images using the location of the object in the one or more images. 
 
     
     
       2. The method of  claim 1 , wherein the attention values are a range of numerical values in a distribution, and wherein the image features are combined using a multiplication operation. 
     
     
       3. The method of  claim 1 , further comprising:
 publishing the one or more modified images on a social network site as an electronic message. 
 
     
     
       4. The method of  claim 1 , wherein the plurality of scaled images comprises a first scaled image and a second scaled image. 
     
     
       5. The method of  claim 4 , wherein the first scaled image is used to generate a first set of attention values and a first image feature dataset. 
     
     
       6. The method of  claim 5 , wherein the second scaled image is used to generate a second set of attention values and a second image feature dataset. 
     
     
       7. The method of  claim 6 , wherein the first set of attention values and the first image feature dataset are multiplied together to produce a first multiplication output; and wherein the second set of attention values and the second image feature dataset are multiplied together to produce a second multiplication output. 
     
     
       8. The method of  claim 7 , further comprising:
 summing the first multiplication output and the second multiplication output to generate a dense feature dataset. 
 
     
     
       9. The method of  claim 8 , wherein the dense feature dataset comprises a plurality of vectors for a plurality of pixels of the image. 
     
     
       10. The method of  claim 1 , wherein the image features and the attention map are generated using one or more convolutional neural networks. 
     
     
       11. The method of  claim 10 , further comprising:
 identifying a source image having a source dense feature; 
 identifying a target image having a target dense feature; and 
 training the one or more convolutional neural networks by at east maximizing a product of the source dense feature and the target dense feature. 
 
     
     
       12. The method of  claim 11 , wherein maximizing the product comprises adjusting parameters in the one or more convolutional networks to maximize the product. 
     
     
       13. The method of  claim 12 , wherein the product is an inner product. 
     
     
       14. A system comprising:
 one or more processors of a machine; and 
 a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising: 
 identify, using one or more processors of a machine, an image; 
 generate a plurality of scaled images from the image; 
 generate image features for the plurality of scaled images; 
 generate a dense features by combining image features with attention values of an attention map, the attention values being one or more numerical values that modify values of the dense features based at least in part on the scale of the plurality of scaled images; 
 identify, using the dense features, a location within one or more images of an object depicted in each of the one or more images; 
 store, in memory, the location of the object within each of the one or more images; and 
 generate one or more modified images from the one or more images using the location of the object in the one or more images. 
 
     
     
       15. The system of  claim 14 , wherein the attention values are a range of numerical values in a distribution, and wherein the image features are combined using a multiplication operation. 
     
     
       16. The system of  claim 14 , the operations further comprising:
 publish the one or more modified images on a social network site as an electronic message. 
 
     
     
       17. The system of  claim 14 , wherein the plurality of scaled images comprises a first scaled image and a second scaled image. 
     
     
       18. The system of  claim 17 , wherein the first scaled image is used to generate a first set of attention values and a first image feature dataset. 
     
     
       19. A non-transitory machine-readable storage device embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
 identify, using one or more processors of a machine, an image; 
 generate a plurality of scaled images from the image; 
 generate image features for the plurality of scaled images; 
 generate a dense features by combining image features with attention values of an attention map, the attention values being one or more numerical values that modify values of the dense features based at least in part on the scale of the plurality of scaled images; 
 identify, using the dense features, a location within one or more images of an object depicted in each of the one or more images; 
 store, in memory, the location of the object within each of the one or more images; and 
 generate one or more modified images from the one or more images using the location of the object in the one or more images. 
 
     
     
       20. The non-transitory machine-readable storage device of  claim 19 , wherein the attention values are a range of numerical values in a distribution, and wherein the image features are combined using a multiplication operation.