Delayed processing for over-delivery determination for content delivery system experimentation

Abstract

A delayed grouping (batch) processing of previous campaign delivery pacing decisions and corresponding outcomes (deliveries) is used to configure a new auction experiment iteration. In the new iteration, a campaign that was previously over-delivered is classified as either (a) over-delivered due to incorrect pacing or (b) over-delivered due to auction experiment design. After the delayed processing, the new auction experiment iteration is conducted with a mitigating action taken on the previously over-delivered campaign if the campaign is classified as (b) over-delivered due to auction experiment design. For example, the mitigating action can include removing the campaign from a subsequent iteration of the experiment, or the experiment can be redesigned. By doing so, the over-delivery caused by the campaign due to the auction experiment design is avoided when performing the new auction experiment iteration.

The invention claimed is: 
     
       1. A method comprising:
 receiving a set of campaign data for content item campaign deliveries made during an auction experiment having a budget-split design or a quota-split design, the set of campaign data containing content from multiple campaigns, including previous campaigns, based on a specified delivery interval where the content was eligible for delivery; 
 based on the set of campaign data, identifying a campaign that was over-delivered during the auction experiment; 
 comparing a pass-through rate for the campaign to a pass-through rate threshold for the campaign; and 
 based on the pass-through rate for the campaign being below the pass-through rate threshold for the campaign, removing the campaign from the auction experiment; 
 wherein the method is performed by one or more computing devices. 
 
     
     
       2. The method of  claim 1 , wherein:
 the auction experiment comprises a plurality of iterations; 
 the set of campaign data is received for content item campaign deliveries made during a first iteration of the plurality of iterations; 
 the campaign is removed from a second iteration of the plurality of iterations; and 
 the second iteration is conducted after the first iteration. 
 
     
     
       3. The method of  claim 1 , further comprising:
 determining a first number of content requests allowed within a budget of the campaign; 
 determining a second number of content requests allowed by an over-budget tolerance; 
 based on the set of campaign data, determining a third number of content requests for which a content item delivery was made; and 
 wherein the identifying the campaign that was over-delivered is based on third number of content requests exceeding a sum of the first number of content requests and the second number of content requests. 
 
     
     
       4. The method of  claim 1 , further comprising:
 determining the pass-through rate threshold for the campaign based on both: 
 
       (a) a remaining budget or quota for the campaign, and (b) a number of content requests for which the campaign was eligible to participate in content item selection events. 
     
     
       5. The method of  claim 1 , wherein the auction experiment comprises a quota-split design. 
     
     
       6. The method of  claim 1 , wherein the auction experiment comprises a budget-split design. 
     
     
       7. A system comprising:
 one or more processors; 
 storage media; and 
 instructions stored in the storage media and which, when executed by the one or more processors, cause the system to perform: 
 receiving, during a batch window, a set of campaign data for content item campaign deliveries made during an auction experiment having a budget-split design or a quota-split design, the set of campaign data containing content from multiple campaigns, including previous campaigns, based on a specified delivery interval where the content was eligible for delivery; 
 after the batch window, identifying one or more campaigns that were over-delivered during the auction experiment; 
 comparing a pass-through rate for an over-delivered campaign, of the one or more over-delivered campaigns, to a pass-through rate threshold for the over-delivered campaign; and 
 based on the pass-through rate for the over-delivered campaign being below the pass-through rate threshold for the over-delivered campaign, removing the over-delivered campaign from at least one iteration of the auction experiment or stopping the auction experiment. 
 
     
     
       8. The system of  claim 7 , further comprising:
 instructions which, when executed by the one or more processors, cause the system to perform: 
 based on the pass-through rate for the over-delivered campaign being below the pass-through rate threshold for the over-delivered campaign, removing the over-delivered campaign from an iteration of the auction experiment. 
 
     
     
       9. The system of  claim 7 , wherein:
 the auction experiment comprises a budget-split design; 
 the budget-split design has a first number of treatment partitions; 
 the system further comprises instructions which, when executed by the one or more processors, cause the system to perform: 
 based on the pass-through rate for the over-delivered campaign being below the pass-through rate threshold for the over-delivered campaign, causing a graphical user interface to be presented at a computing device where the graphical user interface includes graphical user interface controls for reducing a number of treatment partitions of the budget-split design from the first number of treatment partitions. 
 
     
     
       10. The system of  claim 7 , wherein:
 the auction experiment comprises a quota-split design; 
 the quota-split design has a first number of treatment partitions; 
 the system further comprises instructions which, when executed by the one or more processors, cause the system to perform: 
 based on the pass-through rate for the over-delivered campaign being below the pass-through rate threshold for the over-delivered campaign, causing a graphical user interface to be presented at a computing device where the graphical user interface includes graphical user interface controls for reducing a number of treatment partitions of the quota-split design from the first number of treatment partitions. 
 
     
     
       11. The system of  claim 7 , further comprising:
 based on the pass-through rate for the over-delivered campaign being below the pass-through rate threshold for the over-delivered campaign, stopping the auction experiment. 
 
     
     
       12. The system of  claim 7 , further comprising:
 instructions which, when executed by the one or more processors, cause the system to perform: 
 removing the over-delivered campaign from at least one iteration of the auction experiment before a next batch window or stopping the auction experiment before a next batch window. 
 
     
     
       13. The system of  claim 7 , wherein:
 the auction experiment comprises a budget-split design; 
 the budget-split design comprises a plurality of partitions; and 
 the plurality of partitions comprises a control partition and one or more treatment partitions. 
 
     
     
       14. The system of  claim 7 , wherein:
 the auction experiment comprises a quota-split design; 
 the quota-split design comprises a plurality of partitions; and 
 the plurality of partitions comprises a control partition and one or more treatment partitions. 
 
     
     
       15. The system of  claim 7 , wherein:
 the received campaign data comprises, for each campaign of a plurality of campaigns, both: 
 
       (a) a number of content requests remaining within a quota for the campaign and (b) a number of content requests received for which the campaign was eligible;
 the plurality of campaigns includes the over-delivered campaign; 
 the system further comprises instructions which, when executed by the one or more processors, cause the system to perform computing the pass-through rate threshold for the over-delivered campaign based on both: (a) the number of content requests remaining within the quota for the over-delivered campaign and (b) the number of content requests received for which the over-delivered campaign was eligible. 
 
     
     
       16. The system of  claim 7 , wherein:
 the received campaign data comprises, for each campaign of a plurality of campaigns, a number of content item deliveries made for the campaign; 
 the plurality of campaigns includes the over-delivered campaign; and 
 the system the system further comprises instructions which, when executed by the one or more processors, cause the system to perform the identifying the over-delivered campaign based on the number of content item deliveries made for the campaign. 
 
     
     
       17. The system of  claim 7 , wherein:
 the auction experiment encompasses a plurality of budget-split or quota-split partitions; 
 a first partition of the plurality of budget-split or quota-split partitions is assigned to a control arm of the auction experiment; 
 a second partition of the plurality of budget-split or quota-split partitions is assigned to a treatment arm of the auction experiment; 
 during the auction experiment, one or more content requests assigned to the control arm of the auction experiment consume a first portion, of a budget of the over-delivered campaign, allocated to the first partition and do not consume any of the budget of the over-delivered campaign not allocated to the first partition; and 
 during the auction experiment, one or more content requests assigned to the treatment arm of the auction experiment consume a second portion, of the budget of the over-delivered campaign, allocated to the second partition and do not consume any of the budget of the over-delivered campaign not allocated to the second partition. 
 
     
     
       18. The system of  claim 7 , wherein:
 the auction experiment encompasses a bid treatment; and 
 the bid treatment affects what campaigns bid in content request auctions. 
 
     
     
       19. The system of  claim 7 , wherein:
 the auction experiment encompasses a throttling treatment; and 
 the throttling treatment affects which campaigns are called to bid in content request auctions. 
 
     
     
       20. One or more non-transitory computer-readable media comprising:
 instructions which, when executed by one or more computing devices of a computing system, cause the computing system to perform: 
 receiving a set of campaign data for content item campaign deliveries made during an auction experiment having a budget-split design or a quota-split design, the set of campaign data containing content from multiple campaigns, including previous campaigns, based on a specified delivery interval where the content was eligible for delivery; 
 based on the set of campaign data, identifying a campaign that was over-delivered during the auction experiment; 
 comparing a pass-through rate for the campaign to a pass-through rate threshold for the campaign; and 
 based on the pass-through rate for the campaign being below the pass-through rate threshold for the campaign, removing the campaign from the auction experiment.