Method for Human Ranking of Search Results

Abstract

A search engine may perform a search for a user search query. A rankings of result items may be returned as a result of the search. The user may enter new suggested rankings for some of the items. The new suggested rankings may change the rankings of result items when other users search for the same search query at a later time.

1 . A method for human ranking of search engine results, the method performed by a computer system having access to a database, the method comprising:
 receiving at the computer system a search query from a user;   wherein the search query is provided by the user either at the computer system or at a client computing device connected to the computer system over a network;   performing a search over a set of data in response to the search query, the set of data being at the computer system or accessible to the computer system;   generating a rankings of a set of result items returned from the search over the set of data;   adjusting the rankings of the set of result items by a method of using human suggested rankings comprising:
 retrieving from the database a previously suggested rankings of result items that were provided by a other users; 
 wherein the previously suggested rankings of result items contains suggested rankings for zero, one, or more of the result items in the set of result items; 
 wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using the same search query earlier in time; 
 wherein the previously suggested rankings of result items in the database were provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using the same search query earlier in time; 
 adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items; 
   displaying the adjusted rankings of the set of result items in a results document, wherein the user may enter suggested rankings for zero, one, or more result items in the results document, such suggested rankings to be stored in the database to be available for potential later use in adjusting the rankings of a later set of result items returned in response to a later search queries of a later users.   
     
     
         2 . The method of  claim 1 , wherein adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items from the other users to obtain an adjusted rankings of the set of result items is performed by a method comprising: re-ranking one or more result items in the set of result items to the last user-suggested ranking the one or more result items received from the other users, the other users having conducted a search using the same search query earlier in time. 
     
     
         3 . The method of  claim 1 , wherein adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items from the other users to obtain an adjusted rankings of the set of result items is performed by a method comprising: re-ranking one or more result items in the set of result items to an average user-suggested ranking the one or more result items have received from the other users, the other users having conducted a search using the same search query earlier in time. 
     
     
         4 . The method of  claim 1 , wherein adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items from the other users to obtain an adjusted rankings of the set of result items is performed by a method comprising: assigning two weights, the first weight being determined based on user-suggested rankings one or more result items in the set of result items have received from the other users, the other users having conducted a search using the same search query earlier in time, and the second weight being determined by the content, link structure, relevancy, quality, or other substantive qualities of one or more result items in the set of result items, and the two weights being used in combination to adjust the rankings of the one or more result items in the set of result items. 
     
     
         5 . The method of  claim 1 , wherein the results document includes a discussion component for user discussion or a link to a discussion component for user discussion, the discussion component for user discussion comprising a wiki, a forum, a bulletin board, a chat room, or other interface component configured for user discussion. 
     
     
         6 . The method of  claim 1 , wherein the results document includes a version control component that displays information about one or more previous versions of the results document or a link to a version control component that displays information about one or more previous versions of the results document;
 wherein each of the one or more previous versions of the results document display a past adjusted rankings of the set of result items, the past adjusted rankings of the set of result items being generated at the current time or having been generated at a previous time by a method of using human suggested rankings comprising:
 retrieving from the database a previously suggested rankings of result items that were provided by a other users; 
 wherein the previously suggested rankings of result items contains suggested rankings for zero, one, or more of the result items in the set of result items; 
 wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using the same search query earlier in time; 
 wherein the previously suggested rankings of result items in the database were provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using the same search query earlier in time; 
 wherein the other users entered the suggested rankings of result items prior to a specific time, the specific time being associated with the one or more previous versions of the results document; 
 adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items. 
   
     
     
         7 . The method of  claim 6 , wherein the user may choose to view a differences between two or more previous versions of the results document. 
     
     
         8 . The method of  claim 1 , wherein the results document includes user interface elements next to one or more result items;
 wherein the user may enter suggested rankings for the one or more result items via the user interface elements.   
     
     
         9 . The method of  claim 1 , wherein the user may enter suggested rankings for one or more result items listed on the results document by selecting and moving the one or more result items to a higher or lower ranking. 
     
     
         10 . The method of  claim 1 , wherein the user may only enter suggested rankings for the result items in the results document if the user has editing permissions. 
     
     
         11 . The method of  claim 1 , wherein the user must pass a test to show he or she is a human and not a computer program before the user can enter suggested rankings for the result items in the results document. 
     
     
         12 . The method of  claim 1 , wherein the user may only enter suggested rankings for the result items in the results document a limited number of times over a limited time period. 
     
     
         13 . The method of  claim 1 , wherein the search query received from the user is one search query from a set of merged search queries;
 wherein adjusting the rankings of the set of result items by a method of using human suggested rankings is instead performed by a method comprising:
 retrieving from the database a previously suggested rankings of result items that were provided by a other users; 
 wherein the previously suggested rankings of result items contains suggested rankings for zero, one, or more of the result items in the set of result items; 
 wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using any search query from the set of merged search queries earlier in time; 
 wherein the previously suggested rankings of result items in the database were provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using any search query from the set of merged search queries earlier in time; 
 adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items. 
   
     
     
         14 . The method of  claim 13 , wherein the results document includes a merge component that displays information about one or more sets of merged search queries or a link to a merge component that displays information about one or more sets of merged search queries. 
     
     
         15 . The method of  claim 13 , wherein the results document includes user interface elements that may be used to add one or more search queries to a set of merged search queries or to remove one or more search queries from a set of merged search queries. 
     
     
         16 . The method of  claim 1 , wherein the user and the other users are divided into one or more classes;
 wherein the user is in a class;   wherein adjusting the rankings of the set of result items by a method of using human suggested rankings is instead performed by a method comprising:
 retrieving from the database a previously suggested rankings of result items that were provided by a other users; 
 wherein the previously suggested rankings of result items contains suggested rankings for zero, one, or more of the result items in the set of result items; 
 wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using the same search query earlier in time, the other users being in the same class as the user; 
 wherein the previously suggested rankings of result items in the database were provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using the same search query earlier in time; 
 adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items. 
   
     
     
         17 . A system including at least one computer system having access to a database, the system configured to:
 receive at the computer system a search query from a user;   wherein the search query is provided by the user either at the computer system or at a client computing device connected to the computer system over a network;   perform a search over a set of data in response to the search query, the set of data being at the computer system or accessible to the computer system;   generate a rankings of a set of result items returned from the search over the set of data;   adjust the rankings of the set of result items by a method of using human suggested rankings comprising:
 retrieving from the database a previously suggested rankings of result items that were provided by a other users; 
 wherein the previously suggested rankings of result items contains suggested rankings for zero, one, or more of the result items in the set of result items; 
 wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using the same search query earlier in time; 
 wherein the previously suggested rankings of result items in the database were provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using the same search query earlier in time; 
 adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items; 
   display the adjusted rankings of the set of result items in a results document, wherein the user may enter suggested rankings for zero, one, or more result items in the results document, such suggested rankings to be stored in the database for potential later use in adjusting the rankings of a later set of result items returned in response to a later search queries of a later users.   
     
     
         18 . A computer program product, comprising a computer usable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method of human ranking of search engine results, said method using a computer system having access to a database, said method comprising:
 receiving at the computer system a search query from a user;   wherein the search query is provided by the user either at the computer system or at a client computing device connected to the computer system over a network;   performing a search over a set of data in response to the search query, the set of data being at the computer system or accessible to the computer system;   generating a rankings of a set of result items returned from the search over the set of data;   adjusting the rankings of the set of result items by a method of using human suggested rankings comprising:
 retrieving from the database a previously suggested rankings of result items that were provided by a other users; 
 wherein the previously suggested rankings of result items contains suggested rankings for zero, one, or more of the result items in the set of result items; 
 wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using the same search query earlier in time; 
 wherein the previously suggested rankings of result items in the database were provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using the same search query earlier in time; 
 adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items; 
   displaying the adjusted rankings of the set of result items in a results document, wherein the user may enter suggested rankings for zero, one, or more result items in the results document, such suggested rankings to be stored in the database for potential later use in adjusting the rankings of a later set of result items returned in response to a later search queries of a later users.