Image processing apparatus for search of an image, image processing method and storage medium

Abstract

An image processing apparatus is configured to acquire a query image including a subject that is a search target, and an image to be searched, extract a first feature vector that represents features of the search target included in the query image, extract a feature vector map that represents feature vectors of a subject at each position of the image to be searched, perform an arithmetic operation based on the first feature vector and the feature vector map to obtain a heat map that represents the likelihood that the search target is present.

What is claimed is: 
     
       1. An image processing apparatus comprising:
 one or more processors; and 
 one or more memories that store a computer-readable instructions configured to be executed by the one or more processors, thereby the computer-readable instructions causing the image processing apparatus to: 
 acquire a first image including a subject that is a search target, and a second image including a plurality of subjects, one of which may be the search target, from a camera or a storage device; 
 extract a first feature vector that represents features of the search target included in the first image; 
 extract a feature vector map that represents feature vectors of a subject at each position of the second image; 
 perform an arithmetic operation based on the first feature vector and the feature vector map to obtain a heat map that represents the likelihood of a presence of the search target—at each position in the second image; 
 estimate at least one candidate position in which the search target is likely to present in the second image, based on the heat map; 
 extract a second feature vector that represents features of a subject located at the at least one candidate position; 
 identify a position of the search target by determining whether the subject located at the at least one candidate position is the search target based on a similarity between the first feature vector and the second feature vector; and 
 display the position of the search target, 
 wherein when extracting the first feature vector, a feature vector extractor that has been trained in advance is used so that:
 feature vectors that are at a distance less than or equal to a predetermined value from each other are output for images of an identical subject, or subjects that belong to an identical class; and 
 feature vectors that are at a distance greater than or equal to a predetermined value from each other are output for images of non-identical subjects, or subjects that do not belong to an identical class. 
 
 
     
     
       2. The image processing apparatus according to  claim 1 , wherein the subject includes a face. 
     
     
       3. The image processing apparatus according to  claim 1 , wherein when extracting the feature vector map, a feature vector extractor that has been trained in advance is used so that the difference in distance between a feature vector at the position of the subject and the first feature vector in the feature vector map becomes less than or equal to a predetermined value in a case in which the first image and the second image include an identical subject or subjects that belong to an identical class. 
     
     
       4. The image processing apparatus according to  claim 3 , wherein the feature vector extractor is trained based on pairs of an image and a ground truth feature vector map, and wherein when K w  and K h  are constants, the feature vector at the position (x/K w +Δx, y/K h +Δy) of the ground truth feature vector map is calculated by multiplying a feature vector that can be obtained when an image that includes the subject at the position (x, y) of the image is input to the feature vector extractor with a coefficient calculated based on Δx and Δy. 
     
     
       5. The image processing apparatus according to  claim 1 , wherein in the estimation processing, the candidate positions in the second image are estimated based on positions in the heat map in which the value is a maximum and is equal to or greater than a threshold value. 
     
     
       6. The image processing apparatus according to  claim 1 , further configured to:
 acquire a subject image having a higher resolution than the second image, 
 wherein the second feature vector is extracted based on the subject image. 
 
     
     
       7. An image processing method, comprising:
 acquiring a first image including a subject that is a search target, and a second image including a plurality of subjects, one of which may be the search target, from a camera or a storage device; 
 extracting a first feature vector that represents features of the search target included in the first image; 
 extracting a feature vector map that represents feature vectors of a subject at each position of the second image; 
 performing an arithmetic operation based on the first feature vector and the feature vector map to obtain a heat map that represents the likelihood of a presence of the search target at each position in the second image; 
 estimating at least one candidate position in which the search target is likely to present in the second image, based on the heat map; 
 extracting a second feature vector that represents features of a subject located at the at least one candidate position; 
 identifying a position of the search target by determining whether the subject located at the at least one candidate position is the search target based on a similarity between the first feature vector and the second feature vector; and 
 displaying the position of the search target, 
 wherein when extracting the first feature vector, a feature vector extractor that has been trained in advance is used so that:
 feature vectors that are at a distance less than or equal to a predetermined value from each other are output for images of an identical subject, or subjects that belong to an identical class; and 
 feature vectors that are at a distance greater than or equal to a predetermined value from each other are output for images of non-identical subjects, or subjects that do not belong to an identical class. 
 
 
     
     
       8. A non-transitory computer-readable storage medium that stores a program for causing a computer to:
 acquire a first image including a subject that is a search target, and a second image including a plurality of subjects, one of which may be the search target, from a camera or a storage device; 
 extract a first feature vector that represents features of the search target included in the first image; 
 extract a feature vector map that represents feature vectors of a subject at each position of the second image; 
 perform an arithmetic operation based on the first feature vector and the feature vector map to obtain a heat map that represents the likelihood of a presence of the search target at each position in the second image; 
 estimate at least one candidate position in which the search target is likely to present in the second image, based on the heat map; 
 extract a second feature vector that represents features of a subject located at the at least one candidate position; 
 identify a position of the search target by determining whether the subject located at the at least one candidate position is the search target based on a similarity between the first feature vector and the second feature vector; and 
 display the position of the search target, 
 wherein when extracting the first feature vector, a feature vector extractor that has been trained in advance is used so that:
 feature vectors that are at a distance less than or equal to a predetermined value from each other are output for images of an identical subject, or subjects that belong to an identical class; and 
 feature vectors that are at a distance greater than or equal to a predetermined value from each other are output for images of non-identical subjects, or subjects that do not belong to an identical class.