Method and apparatus for evaluating television program recommenders

Abstract

A method and apparatus for validating recommendations generated by a television program recommender uses programmed viewing agents, in which a viewing agent is programmed with a set of rules that characterize the viewing preferences of a modeled viewer. During a training phase, the programmed rules of a viewing agent are applied to a set of training programs to obtain an agent viewing history, which is processed by a profiler to derive an agent profile containing a set of inferred rules. During an evaluation phase, the programmed rules of the viewing agent are applied to test programs to obtain an agent evaluation viewing set. In parallel, the television program recommender generates a set of program recommendations by applying the agent profile to the test programs. The agent evaluation viewing set is then compared with the program recommendations.

What is claimed is:  
     
         1 . A method for validating program recommendations produced by a program recommender, comprising the steps of: 
 generating a viewing agent using one or more programmed rules that characterize viewing preferences;    applying said programmed rules to a set of training programs to obtain an agent viewing history;    processing said agent viewing history with a profiler to derive an agent profile containing a set of inferred rules;    applying said programmed rules to a set of test programs to obtain an agent evaluation viewing set;    generating said program recommendations produced by said program recommender by applying said agent profile to said test programs; and    comparing said agent evaluation viewing set with said program recommendations.    
     
     
         2 . The method of  claim 1 , wherein said viewing agent is selected to have viewing habits that model the viewing habits of a type of viewer.  
     
     
         3 . The method of  claim 1 , wherein said agent viewing history contains an indication of whether said viewing agent liked or disliked each program in a set of training programs.  
     
     
         4 . The method of  claim 1 , wherein said agent evaluation viewing set contains an indication of whether said viewing agent liked or disliked each program in a set of test programs.  
     
     
         5 . The method of  claim 1 , wherein said viewing preferences characterize programs by their attributes that are liked or disliked by said viewing agent.  
     
     
         6 . The method of  claim 1 , wherein said profiler is a component of said program recommender.  
     
     
         7 . The method of  claim 1 , wherein said inferred rules mimic said programmed preferences of the viewing agent.  
     
     
         8 . The method of  claim 1 , wherein said viewing preferences include one or more random programs.  
     
     
         9 . The method of  claim 1 , wherein said viewing preferences change over time.  
     
     
         10 . The method of  claim 1 , wherein said comparing step further comprises the step of comparing calculated precision and recall values to a predefined level of accuracy.  
     
     
         11 . A method for validating program recommendations produced by a program recommender using a viewing agent having programmed viewing preferences, said method comprising the steps of: 
 processing a viewing history of said viewing agent using a profiler to generate an agent profile, said agent profile containing a set of inferred rules that characterize said programmed viewing preferences;    applying said programmed viewing preferences to a set of test programs to obtain an agent evaluation viewing set;    generating said program recommendations produced by said program recommender by applying said agent profile to said test programs; and    comparing said agent evaluation viewing set with said program recommendations.    
     
     
         12 . The method of  claim 11 , wherein said viewing agent is selected to have viewing habits that model the viewing habits of a type of viewer.  
     
     
         13 . The method of  claim 11 , wherein said agent viewing history contains an indication of whether said viewing agent liked or disliked each program in a set of training programs.  
     
     
         14 . The method of  claim 11 , wherein said agent evaluation viewing set contains an indication of whether said viewing agent liked or disliked each program in a set of test programs.  
     
     
         15 . The method of  claim 11 , wherein said viewing preferences characterize programs by their attributes that are liked or disliked by said viewing agent.  
     
     
         16 . The method of  claim 11 , wherein said profiler is a component of said program recommender.  
     
     
         17 . The method of  claim 11 , wherein said inferred rules mimic said programmed preferences of the viewing agent.  
     
     
         18 . The method of  claim 11 , wherein said viewing preferences include one or more random programs.  
     
     
         19 . The method of  claim 11 , wherein said viewing preferences change over time.  
     
     
         20 . The method of  claim 11 , wherein said comparing step further comprises the step of comparing calculated precision and recall values to a predefined level of accuracy.  
     
     
         21 . A method for determining the required size of a viewing history for a program recommender to provide a given level of accuracy, said method comprising the steps of: 
 generating a plurality of viewer agents of varying complexity;    generating viewing histories of varying size for each of said viewing agents;    determining the precision for each viewing history size and viewing agent;    determining the recall for each viewing history size and viewing agent; and    determining a required size for said viewing history such that said precision and recall values exceed a predefined threshold.    
     
     
         22 . The method of  claim 21 , wherein said precision is determined from the true positives (TP) and false positives (FP) as follows:  
       
         
           
             
               Precision 
               = 
               
                 
                   TP 
                   CT 
                 
                 = 
                 
                   TP 
                   
                     TP 
                     + 
                     FP 
                   
                 
               
             
           
           
           
               
           
         
       
     
     
         23 . The method of  claim 21 , wherein said precision is determined from the true positives (TP) and false negatives (FN) as follows:  
       
         
           
             
               Recall 
               = 
               
                 
                   TP 
                   RT 
                 
                 = 
                 
                   TP 
                   
                     TP 
                     + 
                     FN 
                   
                 
               
             
           
           
           
               
           
         
       
     
     
         24 . A method for determining the required size of a viewing history for a program recommender to provide a given level of accuracy for a user, said method comprising the steps of: 
 generating a viewing agent using one or more programmed rules that characterize viewing preferences;    generating a plurality of viewing histories of varying sizes for said viewing agent;    determining the precision for each viewing history;    determining the recall for each viewing history; and    determining a required size for said viewing history such that said precision and recall values exceed a predefined threshold.    
     
     
         25 . A method for determining the required size of a viewing history for a program recommender to provide a given level of accuracy, said method comprising the steps of: 
 generating a plurality of viewer agents with varying program preferences;    generating viewing histories of varying size for each of said viewing agents;    determining the precision for each viewing history size and viewing agent;    determining the recall for each viewing history size and viewing agent; and    determining a required size for said viewing history such that said precision and recall values exceed a predefined threshold.    
     
     
         26 . A system for validating program recommendations produced by a program recommender, comprising: 
 a memory for storing computer readable code; and    a processor operatively coupled to said memory, said processor configured to:    generate a viewing agent using one or more programmed rules that characterize viewing preferences;    apply said programmed rules to a set of training programs to obtain an agent viewing history;    process said agent viewing history with a profiler to derive an agent profile containing a set of inferred rules;    apply said programmed rules to a set of test programs to obtain an agent evaluation viewing set;    generate said program recommendations produced by said program recommender by applying said agent profile to said test programs; and    compare said agent evaluation viewing set with said program recommendations.    
     
     
         27 . A system for validating program recommendations produced by a program recommender using a viewing agent having programmed viewing preferences, comprising: 
 a memory for storing computer readable code; and    a processor operatively coupled to said memory, said processor configured to:    process a viewing history of said viewing agent using a profiler to generate an agent profile, said agent profile containing a set of inferred rules that characterize said programmed viewing preferences;    apply said programmed viewing preferences to a set of test programs to obtain an agent evaluation viewing set;    generate said program recommendations produced by said program recommender by applying said agent profile to said test programs; and    compare said agent evaluation viewing set with said program recommendations.    
     
     
         28 . A system for determining the required size of a viewing history for a program recommender to provide a given level of accuracy, comprising: 
 a memory for storing computer readable code; and    a processor operatively coupled to said memory, said processor configured to:    generate a plurality of viewer agents of varying complexity;    generate viewing histories of varying size for each of said viewing agents;    determine the precision for each viewing history size and viewing agent;    determine the recall for each viewing history size and viewing agent; and    determine a required size for said viewing history such that said precision and recall values exceed a predefined threshold.    
     
     
         29 . A system for determining the required size of a viewing history for a program recommender to provide a given level of accuracy for a user, comprising: 
 a memory for storing computer readable code; and    a processor operatively coupled to said memory, said processor configured to:    generate a viewing agent using one or more programmed rules that characterize viewing preferences;    generate a plurality of viewing histories of varying sizes for said viewing agent;    determine the precision for each viewing history;    determine the recall for each viewing history; and    determine a required size for said viewing history such that said precision and recall values exceed a predefined threshold.    
     
     
         30 . A system for determining the required size of a viewing history for a program recommender to provide a given level of accuracy, comprising: 
 a memory for storing computer readable code; and    a processor operatively coupled to said memory, said processor configured to:    generate a plurality of viewer agents with varying program preferences;    generate viewing histories of varying size for each of said viewing agents;    determine the precision for each viewing history size and viewing agent;    determine the recall for each viewing history size and viewing agent; and    determine a required size for said viewing history such that said precision and recall values exceed a predefined threshold.