Method of evaluating learning rate of recommender systems

Abstract

A recommender system is analyzed to determine various performance characteristics, such as a learning rate for new items, or a learning rate for new subscriber tastes. Comparisons of different recommenders are presented to assist consumers and marketers in selecting appropriate e-commerce sites for purchasing, advertising, etc.

1 . A method of evaluating a recommender system, which recommender is used for recommending items of interest to subscribers of an online content service provider, the method comprising the steps of:
 (a) identifying a first set of reference items to be used in evaluating the recommender system;   (b) generating a plurality of separate recommendations for a second set of items from the recommender system;   (c) correlating said first set of reference items with said second set of items to identify an awareness level exhibited by the recommender system for said first set of reference items;   (d) generating a rating for the recommender system based on the results of step (c) based on an evaluation of said awareness level.   
     
     
         2 . The method of  claim 1  wherein said awareness level is measured by identifying a first number of said first set of reference items which are appear as selections within a second number of said second set of items. 
     
     
         3 . The method of  claim 2 , wherein said awareness level is measured by comparing said first number to said second number. 
     
     
         4 . The method of  claim 1 , wherein said rating is a numerical value. 
     
     
         5 . The method of  claim 1 , wherein at least some of said first set of references items are selected based on a determination of whether the recommender system is likely to have a relatively low number of existing explicit ratings for such first set of reference items. 
     
     
         6 . The method of  claim 1 , further including a step: repeating steps (a) through (d) for a second recommender system, and comparing a second rating for said second recommender system with said rating for the recommender system. 
     
     
         7 . The method of  claim 1 , further including a step: setting up a plurality of proxy accounts and respective demographic profiles prior to step (b) in order to interact with the recommender system. 
     
     
         8 . The method of  claim 1 , further including a step: generating a plurality of separate ratings for the recommender system based on separate evaluations for a plurality of separate demographic groups. 
     
     
         9 . The method of  claim 1 , wherein said first set of reference items are identified by measuring an expected or measured popularity of a particular item. 
     
     
         10 . The method of  claim 9 , wherein said measuring is performed by analyzing activities of a group of individuals interacting with an online website. 
     
     
         11 . The method of  claim 1 , wherein the first set of reference items represent newly released content and the recommender system is tested to determine an extent of an awareness of such newly released content. 
     
     
         12 . The method of  claim 1 , wherein said plurality of separate recommendations are derived from records generated by the recommender system. 
     
     
         13 . The method of  claim 1 , wherein said plurality of separate recommendations are derived from a software agent extracting recommendations from the recommender system. 
     
     
         14 . A method of evaluating a learning rate of a recommender system for new items, which recommender is used for recommending items of interest to subscribers of an online content service provider, the method comprising the steps of:
 (a) identifying a first set of new items to be used in evaluating the recommender system;
 wherein at least some of said first set of new items are characterized by relatively few explicit ratings in a recommender system database; 
   (b) reviewing a plurality of separate recommendations for a second set of items made by the recommender system;   (c) correlating said first set of reference items with said second set of items to identify an awareness level exhibited by the recommender system for said first set of reference items;   (d) generating a rating for the recommender system based on the results of step (c) based on an evaluation of said awareness level.   
     
     
         15 . The method of  claim 14 , wherein at least a subset of said first set of new items are characterized by sparse data in the recommender system database such that fewer than 1% of users identified in said recommender system database have provided an explicit rating for said at subset. 
     
     
         16 . The method of  claim 14 , wherein the recommender system is a collaborative filtering based system. 
     
     
         17 . The method of  claim 14 , wherein the recommender system is a content based filtering system. 
     
     
         18 . A method of evaluating a learning rate of a recommender system for new preferences by users, which recommender is used for recommending items of interest to subscribers of an online content service provider, the method comprising the steps of:
 (a) providing a set of proxy accounts to interact with the recommender system;   (b) identifying a first set of recommendations given by the recommender system to said set of proxy accounts;   (c) modifying a profile of said set of proxy accounts to create a set of modified proxy accounts, including explicit ratings for items which can be recommended by the recommender system;   (d) identifying a second set of recommendations given by the recommender system to said set of modified proxy accounts.   
     
     
         19 . The method of  claim 18 , further including a step: correlating said first set of recommendations with said second set of recommendations to identify an awareness level exhibited by the recommender system for preference changes in said proxy accounts. 
     
     
         20 . The method of  claim 18 , further including a step: generating a rating for the recommender system based on the results of step (d).