Multivariate testing for content discovery systems

Abstract

A system and methods are provided for analyzing user interactions with a media content discovery system using statistical hypothesis testing techniques. As used herein, a media content discovery system generally represents any service that enables users to browse, view, record, purchase, and/or otherwise interact with media content. Examples of media content that may be made available to users via a media content discovery system include, without limitation, movies, television shows, music, etc. Users may interact with a media content discovery system using any of a number of different types of computing devices, including set-top boxes, desktop computers, laptops, handheld devices, game consoles, etc., and over one or more networks, such as the Internet.

What is claimed is: 
     
         1 . A method, comprising:
 receiving first input selecting two or more dimensions associated with a media content discovery system;   for each of the two or more dimensions, receiving second input indicating two or more dimension values;   generating, based on the two or more dimension values for each of the two or more dimensions, a plurality of experiment permutations;   sending, to a media device of a plurality of media devices, one or more instances of each experiment permutation of the plurality of experiment permutations;   for each instance of the one or more instances, determining whether a user takes a particular action associated with the instance.   
     
     
         2 . The method of  claim 1 , further comprising:
 receiving third input selecting an experiment time window;   determining, based on historical user interaction data, whether a threshold number of user interactions are expected to occur during the experiment time window;   in response to determining that a threshold number of user interactions are not expected during the experiment time window, displaying an alert.   
     
     
         3 . The method of  claim 1 , wherein a dimension of the plurality of dimensions corresponds to one or more of: a user interface, a user interface component, a media content item, a media content item type, content item pricing, a media device type, a time of day, a day of the week, an external marketing campaign. 
     
     
         4 . The method of  claim 1 , further comprising receiving input selecting a population of users, wherein each user of the population of users is associated with a media device of the plurality of media devices. 
     
     
         5 . The method of  claim 1 , further comprising receiving input selecting a population of users, the input including specification of one or more user characteristics. 
     
     
         6 . The method of  claim 1 , wherein the action includes one or more of purchasing a media content item, scheduling a recording a media content item, viewing a media content item, clicking on a link. 
     
     
         7 . The method of  claim 1 , further comprising:
 based on at least one dimension of the two or more selected dimensions, retrieving a set of possible dimension values from a data source corresponding to the at least one dimension;   wherein the second input includes a selection of at least one dimension value from the set of possible dimension values.   
     
     
         8 . The method of  claim 1 , further comprising:
 based on at least one dimension of the two or more selected dimensions, retrieving a set of possible dimension values from a data source corresponding to the at least one dimension;   wherein the data source is a media content item database.   
     
     
         9 . The method of  claim 1 , further comprising receiving third input selecting at least one action, the at least one action including the particular action;
 wherein the at least one action includes one or more of: scheduling of a recording, selection of a media content item for viewing, an advertisement click-through, a purchase, and a user registration.   
     
     
         10 . The method of  claim 1 , wherein each experiment permutation of the plurality of experiment permutations corresponds to a particular user interface display. 
     
     
         11 . One or more non-transitory computer-readable storage media, storing instructions, which when executed by one or more processors cause:
 receiving first input selecting two or more dimensions associated with a media content discovery system;   for each of the two or more dimensions, receiving second input indicating two or more dimension values;   generating, based on the two or more dimension values for each of the two or more dimensions, a plurality of experiment permutations;   sending, to a media device of a plurality of media devices, one or more instances of each experiment permutation of the plurality of experiment permutations;   for each instance of the one or more instances, determining whether a user takes a particular action associated with the instance.   
     
     
         12 . The one or more non-transitory storage media of  claim 11 , further comprising:
 receiving third input selecting an experiment time window;   determining, based on historical user interaction data, whether a threshold number of user interactions are expected to occur during the experiment time window;   in response to determining that a threshold number of user interactions are not expected during the experiment time window, displaying an alert.   
     
     
         13 . The one or more non-transitory storage media of  claim 11 , wherein a dimension of the plurality of dimensions corresponds to one or more of: a user interface, a user interface component, a media content item, a media content item type, content item pricing, a media device type, a time of day, a day of the week, an external marketing campaign. 
     
     
         14 . The one or more non-transitory storage media of  claim 11 , further comprising receiving input selecting a population of users, wherein each user of the population of users is associated with a media device of the plurality of media devices. 
     
     
         15 . The one or more non-transitory storage media of  claim 11 , further comprising receiving input selecting a population of users, the input including specification of one or more user characteristics. 
     
     
         16 . The one or more non-transitory storage media of  claim 11 , wherein the action includes one or more of purchasing a media content item, scheduling a recording a media content item, viewing a media content item, clicking on a link. 
     
     
         17 . The one or more non-transitory storage media of  claim 11 , further comprising:
 based on at least one dimension of the two or more selected dimensions, retrieving a set of possible dimension values from a data source corresponding to the at least one dimension;   wherein the second input includes a selection of at least one dimension value from the set of possible dimension values.   
     
     
         18 . The one or more non-transitory storage media of  claim 11 , further comprising:
 based on at least one dimension of the two or more selected dimensions, retrieving a set of possible dimension values from a data source corresponding to the at least one dimension;   wherein the data source is a media content item database.   
     
     
         19 . The one or more non-transitory storage media of  claim 11 , further comprising receiving third input selecting at least one action, the at least one action including the particular action;
 wherein the at least one action includes one or more of: scheduling of a recording, selection of a media content item for viewing, an advertisement click-through, a purchase, and a user registration.   
     
     
         20 . The one or more non-transitory storage media of  claim 11 , wherein each experiment permutations of the plurality of experiment permutations corresponds to a particular user interface display. 
     
     
         21 . An apparatus, comprising:
 a subsystem, implemented at least partially in hardware, that receives first input selecting two or more dimensions associated with a media content discovery system;   a subsystem, implemented at least partially in hardware, that for each of the two or more dimensions, receives second input indicating two or more dimension values;   a subsystem, implemented at least partially in hardware, that generates, based on the two or more dimension values for each of the two or more dimensions, a plurality of experiment permutations;   a subsystem, implemented at least partially in hardware, that sends, to a media device of a plurality of media devices, one or more instances of each experiment permutations of the plurality of experiment permutations;   a subsystem, implemented at least partially in hardware, that for each instance of the one or more instances, determines whether a user takes a particular action associated with the instance.   
     
     
         22 . The apparatus of  claim 21 , further comprising:
 a subsystem, implemented at least partially in hardware, that receives third input selecting an experiment time window;   a subsystem, implemented at least partially in hardware, that determines, based on historical user interaction data, whether a threshold number of user interactions are expected to occur during the experiment time window;   a subsystem, implemented at least partially in hardware, that in response to determining that a threshold number of user interactions are not expected during the experiment time window, displays an alert.   
     
     
         23 . The apparatus of  claim 21 , wherein a dimension of the plurality of dimensions corresponds to one or more of: a user interface, a user interface component, a media content item, a media content item type, content item pricing, a media device type, a time of day, a day of the week, an external marketing campaign. 
     
     
         24 . The apparatus of  claim 21 , further comprising a subsystem, implemented at least partially in hardware, that receives input selecting a population of users, wherein each user of the population of users is associated with a media device of the plurality of media devices. 
     
     
         25 . The apparatus of  claim 21 , further comprising a subsystem, implemented at least partially in hardware, that receives input selecting a population of users, the input including specification of one or more user characteristics. 
     
     
         26 . The apparatus of  claim 21 , wherein the action includes one or more of purchasing a media content item, scheduling a recording a media content item, viewing a media content item, clicking on a link. 
     
     
         27 . The apparatus of  claim 21 , further comprising:
 a subsystem, implemented at least partially in hardware, that based on at least one dimension of the two or more selected dimensions, retrieves a set of possible dimension values from a data source corresponding to the at least one dimension;   wherein the second input includes a selection of at least one dimension value from the set of possible dimension values.   
     
     
         28 . The apparatus of  claim 21 , further comprising:
 a subsystem, implemented at least partially in hardware, that based on at least one dimension of the two or more selected dimensions, retrieves a set of possible dimension values from a data source corresponding to the at least one dimension;   wherein the data source is a media content item database.   
     
     
         29 . The apparatus of  claim 21 , further comprising a subsystem, implemented at least partially in hardware, that receives third input selecting at least one action, the at least one action including the particular action;
 wherein the at least one action includes one or more of: scheduling of a recording, selection of a media content item for viewing, an advertisement click-through, a purchase, and a user registration.   
     
     
         30 . The apparatus of  claim 21 , wherein each experiment permutations of the plurality of experiment permutations corresponds to a particular user interface display.