Identifying objects in images

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for identifying objects in images. One of the methods includes receiving an input image; down-sampling the input image to generate a second image; generating a respective first score for each of the plurality of object categories; selecting an initial patch of the input image; generating a respective second score for each of the plurality of object categories; and generating a respective third score for each of the plurality of object categories from the first scores and the second scores, wherein the respective third score for each of the plurality of object categories represents a likelihood that the input image contains an image of an object belonging to the object category.

What is claimed is: 
     
       1. A method comprising:
 receiving, by an image classification system, an input image having a first resolution; 
 down-sampling, by the image classification system, the input image to generate a second image having a second, lower resolution; 
 selecting, by the image classification system, an initial, first-resolution patch of the input image; 
 processing, by the image classification system, the second image using a low-resolution neural network, wherein the low-resolution neural network is configured to process the second image to generate a plurality of first scores, wherein each first score corresponds to a respective object category from a predetermined set of object categories, and wherein each first score represents a respective likelihood that the second image contains an image of an object belonging to the corresponding object category; 
 processing, by the image classification system, the initial patch of the input image using an initial patch neural network, wherein the initial patch neural network is configured to process the initial patch of the input image to generate a plurality of second scores, wherein each second score corresponds to a respective object category from the predetermined set of object categories, and wherein each second score represents a respective likelihood that the initial patch contains an image of an object belonging to the corresponding object category; and 
 generating, by the image classification system, a respective third score corresponding to each object category from the predetermined set of object categories from the first scores and the second scores, wherein each third score represents a respective likelihood that the input image contains an image of an object belonging to the corresponding object category. 
 
     
     
       2. The method of  claim 1 , wherein generating the third scores comprises:
 for each object category, combining the first score corresponding to the object category and the second score corresponding to the object category to generate the third score corresponding to the object category. 
 
     
     
       3. The method of  claim 2 , wherein combining the first score corresponding to the object category and the second score corresponding to the object category comprises computing a geometric mean of the first score and the second score. 
     
     
       4. The method of  claim 1 , wherein the low-resolution neural network is configured to:
 process the second image to generate a plurality of features of the second image; and 
 process the plurality of features of the second image to generate the plurality of first scores. 
 
     
     
       5. The method of  claim 4 , wherein selecting, by the image classification system, the initial, first-resolution patch of the input image comprises:
 generating coordinates of a reference point in the input image by processing the plurality of features of the second image and the first scores using a patch locator neural network; and 
 selecting a patch of the input image that is centered at the reference point as the initial, first-resolution patch of the input image. 
 
     
     
       6. The method of  claim 4 ,
 wherein the low-resolution neural network comprises a plurality of layers, 
 wherein each of the layers is configured to, during the processing of the second image using the low-resolution neural network, receive a respective input at each of the plurality of layers and producing a respective output based on a set of parameters of the layer, and 
 wherein the features of the second image are the output of a pre-determined layer of the plurality of layers. 
 
     
     
       7. The method of  claim 4 , further comprising:
 processing the features of the second image, the first scores, and the second scores using an additional patch locator neural network to select an additional patch of the input image at the first resolution; and 
 processing the additional patch using an additional patch neural network, wherein the additional patch neural network is configured to process the additional patch to generate a plurality of fourth scores, each fourth score corresponding to a respective object category of the predetermined set of object categories. 
 
     
     
       8. The method of  claim 7 , wherein generating the third scores comprises:
 for each object category, combining the first score corresponding to the object category, the second score corresponding to the object category, and the fourth score corresponding to the object category to generate the third score for the object category. 
 
     
     
       9. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
 receiving, by an image classification system, an input image having a first resolution; 
 down-sampling, by the image classification system, the input image to generate a second image having a second, lower resolution; 
 selecting, by the image classification system, an initial, first-resolution patch of the input image; 
 processing, by the image classification system, the second image using a low-resolution neural network, wherein the low-resolution neural network is configured to process the second image to generate a plurality of first scores, wherein each first score corresponds to a respective object category from a predetermined set of object categories, and wherein each first score represents a respective likelihood that the second image contains an image of an object belonging to the corresponding object category; 
 processing, by the image classification system, the initial patch of the input image using an initial patch neural network, wherein the initial patch neural network is configured to process the initial patch of the input image to generate a plurality of second scores, wherein each second score corresponds to a respective object category from the predetermined set of object categories, and wherein each second score represents a respective likelihood that the initial patch contains an image of an object belonging to the corresponding object category; and 
 generating, by the image classification system, a respective third score corresponding to each object category from the predetermined set of object categories from the first scores and the second scores, wherein each third score represents a respective likelihood that the input image contains an image of an object belonging to the corresponding object category. 
 
     
     
       10. The system of  claim 9 , wherein generating the third scores comprises:
 for each object category, combining the first score corresponding to the object category and the second score corresponding to the object category to generate the third score corresponding to the object category. 
 
     
     
       11. The system of  claim 10 , wherein combining the first score corresponding to the object category and the second score corresponding to the object category comprises computing a geometric mean of the first score and the second score. 
     
     
       12. The system of  claim 9 , wherein the low-resolution neural network is configured to:
 process the second image to generate a plurality of features of the second image; and 
 process the plurality of features of the second image to generate the plurality of first scores. 
 
     
     
       13. The system of  claim 12 , wherein selecting, by the image classification system, the initial, first-resolution patch of the input image comprises:
 generating coordinates of a reference point in the input image by processing the plurality of features of the second image and the first scores using a patch locator neural network; and 
 selecting a patch of the input image that is centered at the reference point as the initial, first-resolution patch of the input image. 
 
     
     
       14. The system of  claim 12 ,
 wherein the low-resolution neural network comprises a plurality of layers, 
 wherein each of the layers is configured to, during the processing of the second image using the low-resolution neural network, receive a respective input at each of the plurality of layers and producing a respective output based on a set of parameters of the layer, and 
 wherein the features of the second image are the output of a pre-determined layer of the plurality of layers. 
 
     
     
       15. The system of  claim 12 , the operations further comprising:
 processing the features of the second image, the first scores, and the second scores using an additional patch locator neural network to select an additional patch of the input image at the first resolution; and 
 processing the additional patch using an additional patch neural network, wherein the additional patch neural network is configured to process the additional patch to generate a plurality of fourth scores, each fourth score corresponding to a respective object category of the predetermined set of object categories. 
 
     
     
       16. The system of  claim 15 , wherein generating the third scores comprises:
 for each object category, combining the first score corresponding to the object category, the second score corresponding to the object category, and the fourth score corresponding to the object category to generate the third score for the object category. 
 
     
     
       17. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
 receiving, by an image classification system, an input image having a first resolution; 
 down-sampling, by the image classification system, the input image to generate a second image having a second, lower resolution; 
 selecting, by the image classification system, an initial, first-resolution patch of the input image; 
 processing, by the image classification system, the second image using a low-resolution neural network, wherein the low-resolution neural network is configured to process the second image to generate a plurality of first scores, wherein each first score corresponds to a respective object category from a predetermined set of object categories, and wherein each first score represents a respective likelihood that the second image contains an image of an object belonging to the corresponding object category; 
 processing, by the image classification system, the initial patch of the input image using an initial patch neural network, wherein the initial patch neural network is configured to process the initial patch of the input image to generate a plurality of second scores, wherein each second score corresponds to a respective object category from the predetermined set of object categories, and wherein each second score represents a respective likelihood that the initial patch contains an image of an object belonging to the corresponding object category; and 
 generating, by the image classification system, a respective third score corresponding to each object category from the predetermined set of object categories from the first scores and the second scores, wherein each third score represents a respective likelihood that the input image contains an image of an object belonging to the corresponding object category. 
 
     
     
       18. The computer storage media of  claim 17 , wherein generating the third scores comprises:
 for each object category, combining the first score corresponding to the object category and the second score corresponding to the object category to generate the third score corresponding to the object category. 
 
     
     
       19. The computer storage media of  claim 18 , wherein the low-resolution neural network is configured to:
 process the second image to generate a plurality of features of the second image; and 
 process the plurality of features of the second image to generate the plurality of first scores. 
 
     
     
       20. The computer storage media of  claim 19 , wherein selecting, by the image classification system, the initial, first-resolution patch of the input image comprises:
 generating coordinates of a reference point in the input image by processing the plurality of features of the second image and the first scores using a patch locator neural network; and 
 selecting a patch of the input image that is centered at the reference point as the initial, first-resolution patch of the input image.