System for effective use of data for personalization

Abstract

Off-policy evaluation of a new “target” policy is performed using historical data gathered based on a previous “logging” policy to estimate the performance of the target policy. An estimator may be used, wherein either a quality-based estimator or a quality-agnostic estimator is used to weight the difference between an observed reward in the historical data and an estimated reward generated by the target policy. A quality-agnostic estimator may be used to evaluate an importance weight according to a threshold. In such examples, when the importance weight exceeds the threshold, the quality-agnostic estimator clips the importance weight at the threshold, thereby providing an fixed upper bound irrespective of the quality of the reward predictor. In other examples, a quality-based estimator is used, in which an upper bound incorporates the quality of the reward predictor in order to modify an importance weight used by the estimator.

What is claimed is: 
     
         1 . A system comprising:
 at least one processor; and   memory storing instructions that, when executed by the at least one processor, causes the system to perform a set of operations, the set of operations comprising:
 generating a reward predictor for historical data associated with a logging policy; 
 determining an off-policy evaluation model, wherein the off-policy evaluation model comprises an estimator selected from the group consisting of a quality-agnostic estimator and a quality-based estimator; 
 evaluating, using the off-policy evaluation model, a target policy to determine whether an expected reward metric of the target policy is higher than a reward metric of the logging policy; and 
 when it is determined that the expected reward metric is higher than the reward metric of the logging policy, generating an indication to use the target policy instead of the logging policy. 
   
     
     
         2 . The system of  claim 1 , wherein determining the off-policy evaluation model comprises:
 generating, for the quality-agnostic estimator, a first mean squared error (MSE) metric;   generating, for the quality-based estimator, a second MSE metric;   when the first MSE is lower than the second MSE, selecting the quality-agnostic estimator as the estimator; and   when the second MSE is lower than the first MSE, selecting the quality-based estimator as the estimator.   
     
     
         3 . The system of  claim 1 , wherein the off-policy evaluation model comprises a combination of direct modeling of a reward predictor and inverse propensity scoring, and wherein a weight of the reward predictor in the off-policy evaluation model is determined according to the estimator. 
     
     
         4 . The system of  claim 1 , wherein determining the off-policy evaluation model comprises determining a hyperparameter for the estimator. 
     
     
         5 . The system of  claim 1 , wherein the set of operations further comprises:
 receiving, from a user device, a second indication of a context;   determining, according to the target policy, an action based on the received context; and   providing, in response to the first indication, a third indication of the determined action.   
     
     
         6 . The system of  claim 1 , wherein the quality-agnostic estimator comprises a threshold at which an importance weight is clipped if the weight exceeds the threshold. 
     
     
         7 . The system of  claim 1 , wherein the set of operations further comprises:
 accessing the historical data from a historical data store, wherein the historical data comprises at least one context, an action associated with the context, and a reward for the action.   
     
     
         8 . A method for selecting a new policy based on a previous policy, the method comprising:
 accessing historical data associated with the previous policy, the historical data comprising at least one context, an action determined based on the context, and a reward for the action;   evaluating, using an off-policy evaluation model, the new policy to determine whether to use the new policy instead of the previous policy, wherein the off-policy evaluation model comprises a combination of a direct model, inverse propensity scoring, and an estimator selected from the group consisting of a quality-agnostic estimator and a quality-based estimator; and   based on determining that the new policy should be used instead of the previous policy:
 generating an action for a context according to the new policy; and 
 providing an indication of the action to a user device. 
   
     
     
         9 . The method of  claim 1 , wherein the new policy is determined to be used instead of the old policy when an average reward metric for the new policy is higher than an average reward metric for previous policy, and wherein the average reward metric for the new policy is determined using the off-policy evaluation model. 
     
     
         10 . The method of  claim 1 , wherein the estimator of the off-policy evaluation model is selected by:
 generating, for the quality-agnostic estimator, a first mean squared error (MSE) metric;   generating, for the quality-based estimator, a second MSE metric;   when the first MSE is lower than the second MSE, selecting the quality-agnostic estimator as the estimator; and   when the second MSE is lower than the first MSE, selecting the quality-based estimator as the estimator.   
     
     
         11 . The method of  claim 10 , further comprising:
 determining, for the selected estimator, a hyperparameter for the estimator, wherein the hyperparameter is determined by iteratively refining the hyperparameter to reduce the MSE of the selected estimator.   
     
     
         12 . The method of  claim 1 , wherein the quality-agnostic estimator comprises a threshold at which an importance weight is clipped if the weight exceeds the threshold. 
     
     
         13 . The method of  claim 1 , wherein the direct model is a regression model for the historical data, and wherein the inverse propensity scoring generates a weight for a predicted reward. 
     
     
         14 . A method for off-policy evaluation of a target policy, the method comprising:
 generating a reward predictor for historical data associated with a logging policy;   determining an off-policy evaluation model, wherein the off-policy evaluation model comprises an estimator selected from the group consisting of a quality-agnostic estimator and a quality-based estimator;   evaluating, using the off-policy evaluation model, the target policy to determine whether an expected reward metric of the target policy is higher than a reward metric of the logging policy; and   when it is determined that the expected reward metric is higher than the reward metric of the logging policy, generating an indication to use the target policy instead of the logging policy.   
     
     
         15 . The method of  claim 14 , wherein determining the off-policy evaluation model comprises:
 generating, for the quality-agnostic estimator, a first mean squared error (MSE) metric;   generating, for the quality-based estimator, a second MSE metric;   when the first MSE is lower than the second MSE, selecting the quality-agnostic estimator as the estimator; and   when the second MSE is lower than the first MSE, selecting the quality-based estimator as the estimator.   
     
     
         16 . The method of  claim 14 , wherein the off-policy evaluation model comprises a combination of direct modeling of a reward predictor and inverse propensity scoring, and wherein a weight of the reward predictor in the off-policy evaluation model is determined according to the estimator. 
     
     
         17 . The method of  claim 14 , wherein determining the off-policy evaluation model comprises generating a hyperparameter for the estimator. 
     
     
         18 . The method of  claim 14 , further comprising:
 receiving, from a user device, a second indication of a context;   determining, according to the target policy, an action based on the received context; and   providing, in response to the first indication, a third indication of the determined action.   
     
     
         19 . The method of  claim 14 , wherein the quality-agnostic estimator comprises a threshold at which an importance weight is clipped if the weight exceeds the threshold. 
     
     
         20 . The method of  claim 14 , further comprising:
 accessing the historical data from a historical data store, wherein the historical data comprises at least one context, an action associated with the context, and a reward for the action.