Generating search requests from multimodal queries

Abstract

A method and system for generating a search request from a multimodal query that includes a query image and query text is provided. The multimodal query system identifies images of a collection that are textually related to the query image based on similarity between words associated with each image and the query text. The multimodal query system then selects those images of the identified images that are visually related to the query image. The multimodal query system may formulate a search request based on keywords of web pages that contain the selected images and submit that search request to a search engine service.

1 . A method in a device for generating a search request for a multimodal query with a query image and query text, the query image being stored in electronic form, the method comprising:
 providing access to a collection of images and associated words;   receiving a multimodal query that includes a query image and query text;   identifying images of the collection based on textual relatedness between a word associated with an image and the query text;   selecting images of the identified images based on visual relatedness between an identified image and the query image;   generating a search request based on keywords associated with the selected images;   submitting the generated search request to a search engine for identifying documents related to the multimodal query; and   providing an indication of the identified documents as a search result for the multimodal query.   
     
     
         2 . The method of  claim 1  wherein the selecting comprises extracting a feature vector for the identified image, determining the distance between the extracted feature vector and the feature vector of each image of the collection, and selecting the images based on the determined distance. 
     
     
         3 . The method of  claim 1  wherein the collection includes a collection of web pages with images and words. 
     
     
         4 . The method of  claim 1  wherein visual relatedness is based on similarity in color space and wavelet coefficients. 
     
     
         5 . The method of  claim 1  including:
 before identifying images of the collection,
 determining whether the query image is a duplicate of an image of the collection; and 
 when the query image is a duplicate of an image, generating a search request based on a keyword associated with that image. 
 
 
     
     
         6 . The method of  claim 5  wherein the query image is a duplicate when the images are identical. 
     
     
         7 . The method of  claim 5  wherein the query image is a duplicate when the images are of the same content but from different points of view. 
     
     
         8 . The method of  claim 5  wherein the collection includes signatures of the images and wherein images are duplicates when they have the same signature. 
     
     
         9 . The method of  claim 1  wherein the query text is derived from audio information. 
     
     
         10 . A computer-readable storage device containing computer-executable instructions for controlling a computing device to find images related to a multimodal query, the instructions for performing a method comprising:
 providing access to web pages with images, the web pages having words;   receiving a query image and query text of the multimodal query;   identifying images of the web pages based on textual relatedness between words of a web page and the query text;   selecting images of the identified images of the web pages based on visual relatedness between an identified image and the query image; and   generating a search request based on keywords associated with the selected images.   
     
     
         11 . The computer-readable storage device of  claim 10  including:
 submitting the generated search request to a search engine for identifying documents related to the multimodal query; and 
 providing an indication of the identified documents as a search result for the multimodal query. 
 
     
     
         12 . The computer-readable storage device of  claim 10  wherein the selecting comprises extracting a feature vector for the query image, determining the distance between the extracted feature vector and the feature vector and the feature vector of each image of the collection, and selecting the images based on the determined distance. 
     
     
         13 . The computer-readable storage device of  claim 10  including:
 before identifying web pages,
 determining whether the query image is a duplicate of an image of a web page; and 
 when the query image is a duplicate of an image of a web page, generating a search request based on words of the web page that contains the duplicate image. 
 
 
     
     
         14 . The computer-readable storage device of  claim 10  wherein visual relatedness is based on similarity in color space and wavelet coefficients. 
     
     
         15 . The computer-readable storage device of  claim 13  wherein the query text is derived from audio information. 
     
     
         16 . A computing device for generating a search request for a multimodal query with a query image and query text, comprising:
 a memory storing computer-executable instructions of:
 a component that identifies images of a collection of images based on textual relatedness between a word associated with an image and the query text; 
 a component that selects images of the identified images based on visual relatedness between an identified image and the query image; and 
 a component that generates a search request based on keywords associated with the selected images; 
   a processor that executes the computer-executable instructions stored in the memory.   
     
     
         17 . The computing device of  claim 16  including:
 a component that submits the generated search request to a search engine for identifying documents related to the multimodal query; and 
 a component that provides an indication of the identified documents as a search result for the multimodal query. 
 
     
     
         18 . The computing device of  claim 17  wherein the component that selects extracts a feature vector for the identified image, determines the distance between the extracted feature vector and the feature vector of each image of the collection, and selects the images based on the determined distance. 
     
     
         19 . The computing device of  claim 17  including a component that before identifying images of the collection, determines whether the query image is a duplicate of an image of the collection and when the query image is a duplicate of an image, generates a search request based on a keyword associated with that image. 
     
     
         20 . The computing device of  claim 17  wherein the collection includes signatures of the images and wherein images are duplicates when they have the same signature.