Methods and apparatus for automatically providing a ranking of items in response to a search request

Abstract

This application relates to apparatus and methods for ranking search results. In some examples, a computing device receives item data identifying items for a search query provided by a user. The computing determines, for each item, a first value based on a relevance of the item to the search query. The computing device also determines user engagement data for the user, where the user engagement data identifies engagements by the user on a webpage. The computing device determines, for each item, a second value based on the user engagement data. In some examples, the computing device determines the first values and second values by executing one or more machine learning models. The computing device further determines a ranking of the plurality of items based on the first values and the second values. The computing may transmit the ranking of the plurality of items, such as to a web server.

What is claimed is: 
     
       1. A system comprising:
 a computing device comprising at least one processor, the at least one processor configured to:
 receive item data identifying a plurality of items for a search query provided by a user; 
 determine, for each of the plurality of items, a first value based on a relevance of each of the plurality of items to the search query; 
 determine user engagement data for the user, wherein the user engagement data identifies engagements by the user for each of the plurality of items on a webpage, wherein the user engagement data identifies at least a first engagement type, a second engagement type, and a third engagement type; 
 determine, for each of the plurality of items, a weight value based on the user engagement data for each of the plurality of items, wherein:
 the weight value is a first weight value when the first engagement type is above a first threshold; 
 the weight value is a second weight value when the first engagement type is not above the first threshold and the second engagement type is above a second threshold; 
 the weight value is a third weight value when the second engagement type is not above the second threshold and the third engagement type is above a third threshold; and 
 the weight value is a fourth weight value when the third engagement type is not above the third threshold; 
 
 determine, for each of the plurality of items, a second value based on the user engagement data and the weight values; 
 determine, based on the first values and the second values, a ranking of the plurality of items; and 
 transmit the ranking of the plurality of items. 
 
 
     
     
       2. The system of  claim 1 , wherein the item data identifies an initial ranking of the plurality of items, and wherein the at least one processor is configured to:
 determine, for each of the plurality of items, a third value based on the initial ranking of the plurality of items and the user engagement data corresponding to each of the plurality of items, wherein determining the second value for each of the plurality of items is based on the third value. 
 
     
     
       3. The system of  claim 2 , wherein the third value for each of the plurality of items is determined based on the corresponding weight value. 
     
     
       4. The system of  claim 3 , wherein the first engagement type comprises a number of clicks, the second engagement type comprises a number of add-to-cart events, and the third engagement type comprises a number of orders, wherein determining the weight value comprises:
 determining the weight value is the first weight value when the number of clicks is above a first threshold; 
 determining the weight value is the second weight value when the number of clicks is not above the first threshold and the number of add-to-cart events is above a second threshold; 
 determining the weight value is the third weight value when the number of add-to-cart events is not above the second threshold and the number of orders is above a third threshold; and 
 determining the weight value is the fourth weight value when the number of orders is not above the third threshold. 
 
     
     
       5. The system of  claim 1 , wherein the first weight value is greater than the second weight value, the second weigh value is greater than the third weight value, and the third weight value is greater than the fourth weight value. 
     
     
       6. The system of  claim 1 , wherein determining, for each of the plurality of items, a first value based on the relevance of each of the plurality of items to the search query comprises applying a machine learning model based on gradient boosted trees to the first values. 
     
     
       7. The system of  claim 6 , wherein the at least one processor is configured to train the machine learning model with crowd model data. 
     
     
       8. The system of  claim 1 , wherein determining, for each of the plurality of items, the second value based on the user engagement data and the weight values comprises applying a machine learning model based on gradient boosted trees to the second values and the weight values. 
     
     
       9. The system of  claim 8 , wherein the at least one processor is configured to train the machine learning model with historical engagement data identifying previous engagements by a plurality of users on the webpage. 
     
     
       10. The system of  claim 1 , wherein the at least one processor is configured to:
 receive the search query from a web server; 
 request the item data identifying the plurality of items from an item recommendation system; and 
 receive the item data from the item recommendation system. 
 
     
     
       11. The system of  claim 1 , wherein transmitting the ranking of the plurality of items comprises transmitting the ranking of the plurality of items to a web server, wherein the web server is configured to display item advertisements for the plurality of items according to the received ranking of the plurality of items. 
     
     
       12. A method comprising:
 receiving item data identifying a plurality of items for a search query provided by a user; 
 determining, for each of the plurality of items, a first value based on a relevance of each of the plurality of items to the search query; 
 determining user engagement data for the user, wherein the user engagement data identifies engagements by the user for each of the plurality of items on a webpage, wherein the user engagement data identifies at least a first engagement type, a second engagement type, and a third engagement type; 
 determining, for each of the plurality of items, a weight value based on the user engagement data for each of the plurality of items, wherein:
 the weight value is a first weight value when the first engagement type is above a first threshold; 
 the weight value is a second weight value when the first engagement type is not above the first threshold and the second engagement type is above a second threshold; 
 the weight value is a third weight value when the second engagement type is not above the second threshold and the third engagement type is above a third threshold; and 
 the weight value is a fourth weight value when the third engagement type is not above the third threshold; 
 
 determining, for each of the plurality of items, a second value based on the user engagement data and the weight values; 
 determining, based on the first values and the second values, a ranking of the plurality of items; and 
 transmitting the ranking of the plurality of items. 
 
     
     
       13. The method of  claim 12  wherein the item data identifies an initial ranking of the plurality of (items, and wherein the method further comprises:
 determining, for each of the plurality of items, a third value based on the initial ranking of the plurality of items and the user engagement data corresponding to each of the plurality of items, wherein determining the second value for each of the plurality of items is based on the third value. 
 
     
     
       14. The method of  claim 13  wherein the third value for each of the plurality of items is determined based on the corresponding weight value. 
     
     
       15. The method of  claim 14 , wherein the first engagement type comprises a number of clicks, the second engagement type comprises a number of add-to-cart events, and the third engagement type comprises a number of orders, wherein determining the weight value comprises:
 determining the weight value is the first weight value when the number of clicks is above a first threshold; 
 determining the weight value is the second weigh value when the number of clicks is not above the first threshold and the number of add-to-cart events is above a second threshold; 
 determining the weight value is the third weight value when the number of add-to-cart events is not above the second threshold and the number of orders is above a third threshold; and 
 determining the weight value is the fourth weight value when the number of orders is not above the third threshold. 
 
     
     
       16. The method of  claim 12  further comprising:
 receiving the search query from a web server; 
 requesting the item data identifying the plurality of items from an item recommendation system; and 
 receiving the item data from the item recommendation system. 
 
     
     
       17. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by at least one processor, cause a device to perform operations comprising:
 receiving item data identifying a plurality of items for a search query provided by a user; 
 determining, for each of the plurality of items, a first value based on a relevance of each of the plurality of items to the search query; 
 determining user engagement data for the user, wherein the user engagement data identifies engagements by the user for each of the plurality of items on a webpage, wherein the user engagement data identifies at least a first engagement type, a second engagement type, and a third engagement type; 
 determining, for each of the plurality of items, a weight value based on the user engagement data for each of the plurality of items, wherein:
 the weight value is a first weight value when the first engagement type is above a first threshold; 
 the weight value is a second weight value when the first engagement type is not above the first threshold and the second engagement type is above a second threshold; 
 the weight value is a third weight value when the second engagement type is not above the second threshold and the third engagement type is above a third threshold; and 
 
 the weight value is a fourth weight value when the third engagement type is not above the third threshold; 
 determining, for each of the plurality of items, a second value based on the user engagement data and the weight values; 
 determining, based on the first values and the second values, a ranking of the plurality of items; and 
 transmitting the ranking of the plurality of items. 
 
     
     
       18. The non-transitory computer readable medium of  claim 17 , wherein the item data identifies an initial ranking of the plurality of items, further comprising instructions stored thereon that, when executed by the at least one processor, further cause the device to perform operations comprising:
 determining, for each of the plurality of items, a third value based on the initial ranking of the plurality of items and the user engagement data corresponding to each of the plurality of items, wherein determining the second value for each of the plurality of items is based on the third value. 
 
     
     
       19. The non-transitory computer readable medium of  claim 18  further comprising instructions stored thereon that, when executed by the at least one processor, further cause the device to perform operations comprising determining the third value for each of the plurality of items based on the corresponding weight value. 
     
     
       20. The non-transitory computer readable medium of  claim 19 , wherein the first engagement type comprises a number of clicks, the second engagement type comprises a number of add-to-cart events, and the third engagement type comprises a number of orders, the non-transitory computer readable medium further comprising instructions stored thereon that, when executed by the at least one processor, further cause the device to perform operations comprising:
 determining the weight value is the first weight value when the number of clicks is above a first threshold; 
 determining the weight value is the second weigh value when the number of clicks is not above the first threshold and the number of add-to-cart events is above a second threshold; 
 determining the weight value is the third weight value when the number of add-to-cart events is not above the second threshold and the number of orders is above a third threshold; and 
 determining the weight value is the fourth weight value when the number of orders is not above the third threshold.