Techniques for identifying color profiles for textual queries

Abstract

The disclosed techniques include at least one computer-implemented method performed by a system. The system can receive a textual query and process query features of the textual query to identify a color profile indicative of a color intent of the query. The system can identify candidate images that at least partially match the desired content and color intent of the query. The system can further order candidate images based in part on a similarity of a candidate color profile for each candidate image with the identified color profile of the query, and output image data indicative of the ordered set of candidate images.

We claim: 
     
       1. A computer-implemented method comprising:
 receiving a query that contains text including one or more terms that indicate a desired content for an image; 
 generating query features based on the one or more terms of the query; 
 generating a color profile comprising a color histogram indicative of a color intent from the one or more terms of the query by utilizing a machine learning model to process the query features of the one or more terms according to learned color profiles of a plurality of previous queries; 
 identifying a set of candidate images associated with respective candidate color profiles, each of the set of candidate images at least partially matching the desired content and the color intent from the one or more terms of the query; 
 ordering the set of candidate images based in part on a similarity of the candidate color profile for each candidate image with the color profile of the query; and 
 outputting image data indicative of the ordered set of candidate images. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein receiving the query comprises utilizing a search engine that implements the machine learning model, to output the image data indicative of the ordered set of candidate images by:
 causing output of the ordered set of candidate images on a display device of a computing device as search results for the query, the query being submitted to the search engine through the computing device. 
 
     
     
       3. The computer-implemented method of  claim 1 , wherein generating the query features based on the one or more terms of the query comprises:
 generating word embeddings for the one or more terms of the query, wherein the query features are based on the word embeddings. 
 
     
     
       4. The computer-implemented method of  claim 3 , wherein utilizing the machine learning model to process the query features comprises:
 processing the word embeddings with a bidirectional long short-term memory layer. 
 
     
     
       5. The computer-implemented method of  claim 3 , wherein the query features are generated by concatenating the word embeddings for the one or more terms of the query. 
     
     
       6. The computer-implemented method of  claim 1 , wherein the query is further processed by a normalized exponential function to generate the color profile. 
     
     
       7. The computer-implemented method of  claim 1 , wherein generating the color profile comprises generating the color histogram including a probability distribution over a plurality of color bins. 
     
     
       8. The computer-implemented method of  claim 1 , wherein the machine learning model includes at least one fully connected layer and a rectifier linear unit. 
     
     
       9. The computer-implemented method of  claim 1 , wherein ordering the set of candidate images comprises:
 ranking the set of candidate images based on an amount of content of each candidate image that matches the desired content, the ranking being weighted by the similarity of the candidate color profile for each candidate image with the color profile of the query. 
 
     
     
       10. The computer-implemented method of  claim 1 , wherein generating the color profile comprises utilizing the machine learning model to process the query features based on a plurality of queries, sets of candidate images, and user interaction features associated with the sets of candidate images. 
     
     
       11. The computer-implemented method of  claim 10 , wherein utilizing the machine learning model to process the query features comprises utilizing click through data for the plurality of queries, the click through data comprising the user interaction features indicating a user selection or non-selection of one or more candidate images. 
     
     
       12. The computer-implemented method of  claim 10 , wherein utilizing the machine learning model to process the query features comprises utilizing click through data for the plurality of queries, the click through data comprising the user interaction features indicating user inputs causing scrolling through a particular set of candidate images. 
     
     
       13. The computer-implemented method of  claim 10 , wherein utilizing the machine learning model to process the query features comprises utilizing click through data for the plurality of queries, the click through data comprising the user interaction features indicating a sequence of candidate images selected by a particular user. 
     
     
       14. The computer-implemented method of  claim 10 , wherein utilizing the machine learning model to process the query features comprises utilizing click through data for the plurality of queries, the click through data comprising the user interaction features including a first type of interaction feature having a first weight of a first magnitude and a second type of interaction feature having a second weight of a second magnitude less than the first magnitude, and wherein the machine learning model is differentially biased based on the first and second magnitudes of the first and second types of interaction features. 
     
     
       15. The computer-implemented method of  claim 14 , wherein the first type of interaction feature is a click action on a candidate image and the second type of interaction feature is a hovering action over a candidate image. 
     
     
       16. The computer-implemented method of  claim 1 , wherein generating the color profile comprises utilizing the machine learning model to process the query features based on user interaction features of a particular user with sets of candidate images for respective queries. 
     
     
       17. The computer-implemented method of  claim 1  further comprising:
 identifying a particular image retrieved by a search engine as a search result for a given query, the particular image being associated with at least one of a caption or a tag; 
 extracting one or more word embeddings from the caption or the tag; 
 processing the particular image with a residual network and histogram to generate one or more image embeddings; 
 generating image features by concatenating the one or more word embeddings with the one or more image embeddings; 
 generating query-image features based on the color profile, the query features, and the image features; and 
 processing the query-image features with a machine learning process to generate a relevance measure for the particular image. 
 
     
     
       18. The computer-implemented method of  claim 17 , the machine learning process comprises:
 a series of machine learning layers including a plurality of fully connected layers and a plurality of rectifier units. 
 
     
     
       19. A computing system comprising:
 a processor; and 
 memory containing instructions that, when executed by the processor, cause the computing system to: 
 receive a query that contains text including one or more terms that indicate a desired content for an image, the query being received by a search engine; 
 generate query features based on the one or more terms of the query; 
 determine, by processing the query features of the one or more terms utilizing a machine learning model, a color histogram indicative of a color intent from the one or more terms of the query to determine a predicted color profile according to learned color profiles of a plurality of query features of a plurality of queries having associated color histograms color, the plurality of queries previously processed by the search engine; 
 identify a set of candidate images that are each associated with respective candidate color histograms, each of the set of candidate images at least partially matching the desired content and the color histogram of the query; 
 ranking the set of candidate images based in part on a similarity of the candidate color histogram of each candidate image with the color histogram of the query; and 
 causing, on a display device, output of results that satisfy the query, the results including the ranked set of candidate images. 
 
     
     
       20. A non-transitory computer-readable medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising:
 generating query features based on text including terms of a query; 
 generating a color profile comprising a color histogram indicative of a color intent from the terms of the query by utilizing a machine learning model to process the query features of the terms according to learned color profiles of a plurality of previous queries; 
 identifying a set of candidate images comprising respective candidate color profiles that at least partially match the color intent from the terms of the query, the set of candidate images being identified from among a plurality of images available to a search engine; 
 ordering the set of candidate images based on a similarity of the candidate color profile for each candidate image with the color profile of the query; and 
 providing image data indicative of the ordered set of candidate image.