Optimization of send time of messages

Abstract

Introduced here are approaches for identifying the optimal send time for messages by accounting for hidden confounders, such as the content of those messages, delivery channel, etc. These approaches use a causal inference framework to discover and then remove the impact of hidden confounders. These approaches may be employed by a marketing and analytics platform (or simply “marketing platform”) that may be used to design, implement, or review digital marketing campaigns. The marketing platform can consider the send time as a treatment and then employ machine learning (ML) models that consider the send time, features of the recipient, and hidden confounders to produce a ranked series of send times with the effect of the hidden confounders marginalized. Approaches to performing offline evaluations that mimic A/B tests using data related to existing field experiments are also introduced here.

1 . A non-transitory computer-readable medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising:
 defining a first treatment group that includes recipients of messages who adopted treatments recommend by a first machine learning model;   defining a second treatment group that includes recipients who adopted treatments recommended by a second machine learning model;   determining a first performance metric for the first treatment group;   determining a second performance metric for the second treatment group; and   establishing which of the first and second machine learning models performed better by comparing the first and second performance metrics.   
     
     
         2 . The non-transitory computer-readable medium of  claim 1 , wherein the first performance metric is determined based on information regarding behaviors of the first treatment group that is derived from a first dataset. 
     
     
         3 . The non-transitory computer-readable medium of  claim 1 , wherein the second performance metric is determined based on information regarding behaviors of the second treatment group that is derived from a first dataset. 
     
     
         4 . The non-transitory computer-readable medium of  claim 1 , wherein the first and second performance metrics are response rate to the messages. 
     
     
         5 . The non-transitory computer-readable medium of  claim 1 , the operations further comprising:
 obtaining a first dataset related to a field experiment in which messages were sent to recipients as part of a digital marketing campaign;   obtaining a second dataset that includes treatments recommended for the recipients by the first machine learning model; and   obtaining a third dataset that includes treatments recommended for the recipients by the second machine learning model.   
     
     
         6 . The non-transitory computer-readable medium of  claim 5 , the operations further comprising cleaning the first dataset to reduce bias by downsampling intervals of time into which the messages can be sorted. 
     
     
         7 . The non-transitory computer-readable medium of  claim 5 , the operations further comprising cleaning the second and third datasets to reduce bias by assigning a weight to each recommendation associated with each recipient included in the first and second treatment groups. 
     
     
         8 . The non-transitory computer-readable medium of  claim 7 , wherein the weight is inversely proportional to a number of recommendations associated with each recipient. 
     
     
         9 . The non-transitory computer-readable medium of  claim 5 , wherein the first and second treatment groups collectively represent a subset of the recipients who received messages as part of the digital marketing campaign. 
     
     
         10 . A method comprising:
 defining a first treatment group that includes recipients of messages who adopted treatments recommend by a first machine learning model;   defining a second treatment group that includes recipients who adopted treatments recommended by a second machine learning model;   determining a first performance metric for the first treatment group;   determining a second performance metric for the second treatment group; and   establishing which of the first and second machine learning models performed better by comparing the first and second performance metrics.   
     
     
         11 . The method of  claim 10 , wherein:
 the first performance metric is determined based on information regarding behaviors of the first treatment group that is derived from a first dataset; and   the second performance metric is determined based on information regarding behaviors of the second treatment group that is derived from a first dataset.   
     
     
         12 . The method of  claim 10 , wherein the first and second performance metrics are response rate to the messages. 
     
     
         13 . The method of  claim 10 , further comprising:
 obtaining a first dataset related to a field experiment in which messages were sent to recipients as part of a digital marketing campaign;   obtaining a second dataset that includes treatments recommended for the recipients by the first machine learning model; and   obtaining a third dataset that includes treatments recommended for the recipients by the second machine learning model.   
     
     
         14 . The method of  claim 13 , further comprising cleaning the first dataset to reduce bias by downsampling intervals of time into which the messages can be sorted. 
     
     
         15 . The method of  claim 14 , further comprising cleaning the second and third datasets to reduce bias by assigning a weight to each recommendation associated with each recipient included in the first and second treatment groups. 
     
     
         16 . The method of  claim 15 , wherein the weight is inversely proportional to a number of recommendations associated with each recipient. 
     
     
         17 . The method of  claim 13 , wherein the first and second treatment groups collectively represent a subset of the recipients who received messages as part of the digital marketing campaign. 
     
     
         18 . A system comprising:
 one or more memory devices; and   one or more processing devices coupled to the one or more memory devices, the one or more processing devices configured to perform operations comprising:
 classifying each recipient who adopted a corresponding treatment in a first series of treatments as a member of a first treatment group; 
 classifying each recipient who adopted a corresponding treatment in a second series of treatments as a member of a second treatment group; 
 calculating a first performance metric for the first treatment group based on behavior of the members of the first treatment group as determined from a first dataset; 
 calculating a second performance metric for the second treatment group based on behavior of the members of the second treatment group as determined from the first dataset; and 
 storing the first and second performance metrics in a data structure representative of a profile. 
   
     
     
         19 . The system of  claim 18 , wherein the operations further comprise outputting for display a visualization component that includes the first and second performance metrics or information indicative of the first and second performance metrics. 
     
     
         20 . The system of  claim 18 , wherein the first and second treatment groups collectively represent a subset of the series of recipients, and, wherein the profile is associated with a first model, a second model, or the first and second models.