Method of testing online recommender system

Abstract

A recommender system can be analyzed to determine various performance characteristics of an online content service provider. The recommender system is analyzed using a predetermined policy, to determine if it satisfies such policy, and/or has other measurable intended and/or unintended biases. The policy can include such parameters as whether a particular profile is presented with certain particular types of items by the recommender system. The reliability of search engines can also be tested using a similar approach.

What is claimed is: 
     
         1 - 25 . (canceled) 
     
     
         26 . An automated method of evaluating recommendations made by a recommender system, which recommender includes one or more programs executing on a server computing machine and is used for recommending items of interest to online users, the method comprising the steps of:
 (a) setting up a policy on a client computing device which is to be verified by testing outputs of the recommender system;   (b) reviewing an actual set of recommendations for items made by the recommender system to a plurality of separate online users;   (c) with the client computing device comparing said actual set of recommendations for items against a target set of recommendations for items associated with said policy;   (d) with the client computing device calculating a compliance level by the recommender system with said policy based on said actual set of recommendations.   
     
     
         27 . The method of  claim 26 , wherein said target set of recommendations for items are based on items recommended by a second separate reference recommender system. 
     
     
         28 . The method of  claim 26 , further including a step of: tabulating the results of step (c) and posting the same on an Internet website, so that a comparison of recommendations made by a plurality of recommender systems and their associated compliance levels is identified. 
     
     
         29 . The method of  claim 28 , wherein a relative bias rating is identified for each recommender system to indicate a deviation from an expected neutral recommendation. 
     
     
         30 . The method of  claim 28 , wherein a relative ratio of relevant recommendations is identified for each recommender system. 
     
     
         31 . The method of  claim 26  wherein the client computing device performs said reviewing step (b) based on analyzing logs of user interactions with said recommender system. 
     
     
         32 . The method of  claim 26  wherein said policy is implemented at the recommender system to bias recommendations for certain types of items. 
     
     
         33 . The method of  claim 26  wherein said policy is implemented at the recommender system to bias recommendations for certain items originating from a particular source. 
     
     
         34 . The method of  claim 26  wherein said policy is implemented at the recommender system to bias recommendations of item only for certain types of users having a target profile. 
     
     
         35 . The method of  claim 26  wherein said plurality of users includes users having different demographic profiles. 
     
     
         36 . The method of  claim 26  wherein a plurality of separate dummy accounts are set up and used for the plurality of separate online users during step (b) to solicit recommendations from the recommender system. 
     
     
         37 . The method of  claim 26  further including a step: generating a notification with the client computing device to an operator of the recommender system in response to a determination that it is not operating in accordance with said policy. 
     
     
         38 . The method of  claim 26  wherein said recommender system is part of a search engine. 
     
     
         39 . An automated method of evaluating recommendations made by a recommender system, which recommender includes one or more programs executing on a computing machine and is used for recommending items of interest to online users, the method comprising the steps of:
 (a) setting up a policy on the recommender system, which policy is a programmed bias to be given to a set of preference items when responding to a user request for items of a first type from the recommender system;   (b) with a client computing device reviewing data logs including an actual set of recommendations of items of said first type made by the recommender system in response to a plurality of item requests;   (c) with the client computing device comparing said actual set of recommendations for items of said first type made in response to said plurality of item requests to determine an overlap with an expected set of recommendations of items of said first type that is based on said policy favoring said set of preference items;   (d) generating a report with the client computing device measuring a compliance level of the recommender system with said policy for items of said first type based on results of step (c).   
     
     
         40 . The method of  claim 39 , further including a step of: tabulating the results of step (c) and posting the same on an Internet website, so that a comparison of recommendations made by a plurality of recommender systems and their associated compliance levels is identified. 
     
     
         41 . The method of  claim 40 , wherein a relative bias rating is identified for each recommender system to indicate a deviation from an expected neutral recommendation. 
     
     
         42 . The method of  claim 40 , wherein a relative ratio of relevant recommendations is identified for each recommender system. 
     
     
         43 . The method of  claim 40  wherein said policy is implemented at the recommender system to bias recommendations for certain types of items. 
     
     
         44 . The method of  claim 39  wherein said policy is implemented at the recommender system to bias recommendations for certain items originating from a particular source. 
     
     
         45 . The method of  claim 39  wherein said policy is implemented at the recommender system to bias recommendations of item only for certain types of users having a target profile. 
     
     
         46 . The method of  claim 39  wherein said plurality of item requests are from users having different demographic profiles. 
     
     
         47 . The method of  claim 39  wherein a plurality of separate dummy accounts are used by the client computing device for the plurality of separate online users during step (b) to solicit recommendations from the recommender system. 
     
     
         48 . The method of  claim 40  further including a step: generating a notification with the client computing device to an operator of the recommender system in response to a determination that it is not operating in accordance with said policy. 
     
     
         49 . The method of  claim 40  wherein said recommender system is part of a search engine.