Systems and methods for image-feature-based recognition

Abstract

Methods and systems are described herein that allow a user to capture a single image snapshot from video, print, or the world around him or her, and obtain additional information relating to the media itself or items of interest displayed in the snapshot. A fingerprint of the snapshot is used as a query and transmitted to the server. Image Feature-Based Recognition, as described herein, uses a feature index to identify a smaller set of candidate matches from a larger database of images based on the fingerprint. Novel methods and systems using a distance metric and a radical hash table design exploit probabilistic effects and allow distinct image features to be preferred over redundant ones, allowing only the more distinctive data points to remain resident within the index, yielding a lean index that can be quickly used in the identification process.

1 .- 19 . (canceled) 
     
     
         20 . A method, comprising:
 identifying a reference image stored in a reference database, identification comprising:
 matching a first query fingerprint with a candidate fingerprint using, at least in part, a first image feature index generated for the reference database; and 
 identifying the reference image based on an association between the candidate fingerprint and the reference image; 
   during identification of the reference image, obtaining a second query fingerprint associated with a second user device and matching the first query fingerprint with the second query fingerprint; and   in response to identification of the reference image and based at least in part on the match between the first query fingerprint and the second query fingerprint, providing additional information associated with the reference image to the second device for displaying to a second user.   
     
     
         21 . The method of  claim 20 , wherein:
 the second query fingerprint is generated from a second query image;   the method further comprises determining that the reference image matches the second query image; and   the additional information is provided to the second device based in part on the determination.   
     
     
         22 . The method of  claim 20 , wherein:
 determining that the reference image matches the second query image further comprises determining a pose of the reference image relative to the second query image; and   the pose is provided to the second device based in part on the determination.   
     
     
         23 . The method of  claim 20 , wherein:
 the first query fingerprint is associated with a first query;   the method further comprises registering the second user device or the second user with the first query; and   the additional information is provided to all users or devices registered with the first query.   
     
     
         24 . The method of  claim 20 , wherein:
 the method further comprises:
 storing, in a cache database, the reference image; 
 obtaining a third query fingerprint; 
 identifying the reference image using the first query fingerprint and the third query fingerprint; and 
 in response to identifying the reference image, providing the additional information to a third device associated with the third query fingerprint. 
   
     
     
         25 . The method of  claim 24 , wherein:
 identifying the reference image using the first query fingerprint and the third query fingerprint comprises matching the first query fingerprint with the third query fingerprint using the multi-part search:
 the first part of the multi-part search including obtaining the first query fingerprint using a second image feature index generated for the cache database; and 
 a second part of the multi-part search including determining a first query image matches a third query image, the first query fingerprint being generated from the first query image and the third query fingerprint being generated from a third query image. 
   
     
     
         26 . The method of  claim 20 , wherein:
 obtaining the second query fingerprint associated with a second user device comprises:
 receiving a second query from the second user device, the second querying including the second query fingerprint. 
   
     
     
         27 . The method of  claim 20 , wherein:
 in response to identification of the reference image and based at least in part on the match between the first query fingerprint and the second query fingerprint, the reference image is provided to the second device for displaying to a second user.   
     
     
         28 . The method of  claim 20 , wherein:
 in response to identification of the reference image and based at least in part on the match between the first query fingerprint and the second query fingerprint, an annotated version of the reference image is provided to the second device for displaying to a second user.   
     
     
         29 . The method of  claim 28 , wherein:
 the annotated version of the reference image includes selectable tags linking to portions the additional information.   
     
     
         30 . A non-transitory, computer-readable medium containing instructions that, when executed by at least one processor of a system, cause the system to perform operations comprising:
 identifying a reference image stored in a reference database, identification comprising:
 matching a first query fingerprint with a candidate fingerprint using, at least in part, a first image feature index generated for the reference database; and 
 identifying the reference image based on an association between the candidate fingerprint and the reference image; 
   during identification of the reference image, obtaining a second query fingerprint associated with a second user device and matching the first query fingerprint with the second query fingerprint; and   in response to identification of the reference image and based at least in part on the match between the first query fingerprint and the second query fingerprint, providing additional information associated with the reference image to the second device for displaying to a second user.   
     
     
         31 . The non-transitory, computer-readable medium of  claim 30 , wherein:
 the second query fingerprint is generated from a second query image;   the operations further comprise determining that the reference image matches the second query image; and   the additional information is provided to the second device based in part on the determination.   
     
     
         32 . The non-transitory, computer-readable medium of  claim 30 , wherein:
 determining that the reference image matches the second query image further comprises determining a pose of the reference image relative to the second query image; and   the pose is provided to the second device based in part on the determination.   
     
     
         33 . The non-transitory, computer-readable medium of  claim 30 , wherein:
 the first query fingerprint is associated with a first query;   the operations further comprise registering the second user device or the second user with the first query; and   the additional information is provided to all users or devices registered with the first query.   
     
     
         34 . The non-transitory, computer-readable medium of  claim 30 , wherein:
 the operations further comprise:
 storing, in a cache database, the reference image; 
 obtaining a third query fingerprint; 
 identifying the reference image using the first query fingerprint and the third query fingerprint; and 
 in response to identifying the reference image, providing the additional information to a third device associated with the third query fingerprint. 
   
     
     
         35 . The non-transitory, computer-readable medium of  claim 34 , wherein:
 identifying the reference image using the first query fingerprint and the third query fingerprint comprises matching the first query fingerprint with the third query fingerprint using the multi-part search:
 the first part of the multi-part search including obtaining the first query fingerprint using a second image feature index generated for the cache database; and 
 a second part of the multi-part search including determining a first query image matches a third query image, the first query fingerprint being generated from the first query image and the third query fingerprint being generated from a third query image. 
   
     
     
         36 . The non-transitory, computer-readable medium of  claim 30 , wherein:
 obtaining the second query fingerprint associated with a second user device comprises:
 receiving a second query from the second user device, the second querying including the second query fingerprint. 
   
     
     
         37 . The non-transitory, computer-readable medium of  claim 30 , wherein:
 in response to identification of the reference image and based at least in part on the match between the first query fingerprint and the second query fingerprint, the reference image is provided to the second device for displaying to a second user.   
     
     
         38 . The non-transitory, computer-readable medium of  claim 30 , wherein:
 in response to identification of the reference image and based at least in part on the match between the first query fingerprint and the second query fingerprint, an annotated version of the reference image is provided to the second device for displaying to a second user.   
     
     
         39 . The non-transitory, computer-readable medium of  claim 38 , wherein:
 the annotated version of the reference image includes selectable tags linking to portions the additional information.