Evaluating modifications to features used by machine learned models applied by an online system

Abstract

An online system identifies an additional feature to evaluate for inclusion in a machine learned model. The additional feature is based on characteristics of one or more dimensions of information maintained by the online system. To generate data for evaluating the additional feature, the online system generates various partitions of stored data, where each partition includes characteristics associated with one or more dimensions on which the additional feature is based. Using values of characteristics in a partition, the online system generates values for the additional feature and includes the values of the additional feature in the partition. Values for the additional feature are generated for various partitions based on the values of characteristics in each partition. The online system combines multiple partitions that include values for the additional feature to generate a training set for evaluating a machine learned model including the additional feature.

1 - 20 . (canceled) 
     
     
         21 . A system, comprising:
 a processor; and   a memory storing instructions, which when executed by the processor, causes the processor to:
 receive data comprising one or more characteristics of content; 
 generate a plurality of partitions of the data, wherein each partition of the plurality of partitions of the data:
 comprises values for the one or more characteristics of content, and 
 corresponds to a respective table that identifies the values of the one or more characteristics of the content; 
 
 update each partition of data to comprise one or more values of at least one additional feature, wherein the one or more values for at least one additional feature is determined, for each example in a training data set, from values of one or more characteristics of the content on which the at least one additional feature is based and that are included in the partition of data; and 
 update the training data set to include the one or more values of the additional feature for each example in the training data set, wherein the updated training data used to train a machine learning model. 
   
     
     
         22 . The system of  claim 21 , wherein the at least one additional feature is identified for the machine learning model for a specified time. 
     
     
         23 . The system of  claim 22 , wherein the at least one additional feature is dynamic and the one or more values for the at least one additional feature is based on values of the one or more characteristics of content maintained by a computing system at a time prior to the specified time, wherein the content maintained by the computing system is selected from a group comprising at least one of a user or a content item. 
     
     
         24 . The system of  claim 22 , wherein the processor is further caused to:
 compute the one or more values of the at least one additional feature based on the partition of data comprising values of one or more characteristics of content associated with a time prior to the specified time.   
     
     
         25 . The system of  claim 22 , wherein updating each partition of data further comprises:
 modifying the partition of data to include a plurality of values for the at least one additional feature, each of the plurality of values for the at least one additional feature associated with a different time.   
     
     
         26 . The system of  claim 21 , wherein the one or more values for at least one additional feature is further determined, for each example in a training data set, from values of one or more characteristics of the content on which the additional feature is based and that are associated with a time prior to a current time. 
     
     
         27 . The system of  claim 21 , wherein the processor is further caused to:
 generate an updated machine learning model based on the updated training data set.   
     
     
         28 . The system of  claim 21 , wherein updating each partition of data further comprises:
 modifying a plurality of partitions of data, in parallel, to include one or more values for the at least one additional feature.   
     
     
         29 . The system of  claim 21 , wherein updating each partition of data further comprises:
 generating a value for the at least one additional feature associated with a time by applying one or more additional machine learning models to one or more values of one or more characteristics of the content, wherein the one or more additional machine learning models account for at least one of a decay rate or a propagation delay.   
     
     
         30 . The system of  claim 21 , wherein the processor is further caused to:
 generate, using the updated training data set, one or more alternative results; and   replace the trained machine learning model with a replacement machine learning model, the replacement machine learning model based on the one or more alternative results.   
     
     
         31 . A method, comprising:
 receiving, by a processor, data comprising one or more characteristics of content;   generating a plurality of partitions of the data, wherein each partition of the plurality of partitions of the data:
 comprises values for the one or more characteristics of content, and 
 corresponds to a respective table that identifies the values of the one or more characteristics of the content; 
   updating each partition of data to comprise one or more values of at least one additional feature, wherein the one or more values for at least one additional feature is determined, for each example in a training data set, from values of one or more characteristics of the content on which the additional feature is based and that are included in the partition of data; and   updating the training data set to include the one or more values of the additional feature for each example in the training data set to train a machine learning model.   
     
     
         32 . The method of  claim 31 , wherein the at least one additional feature is identified for the machine learning model for a specified time. 
     
     
         33 . The method of  claim 32 , wherein the at least one additional feature is dynamic and the value for the at least one additional feature is based on values of the one or more characteristics of content maintained by a computing system at a time prior to the specified time, wherein content maintained by the computing system is selected from a group comprising at least one of a user or a content item. 
     
     
         34 . The method of  claim 32 , further comprising:
 computing the one or more values of the at least one additional feature based on the partition of data comprising values of one or more characteristics of content associated with a time prior to the specified time;   modifying the partition of data to include a plurality of values for the at least one additional feature, each of the plurality of values for the at least one additional feature associated with a different time; and   generating an updated machine learning model based on the updated training data set.   
     
     
         35 . The method of  claim 31 , wherein the one or more values for at least one additional feature is further determined, for each example in a training data set, from values of one or more characteristics of the content on which the at least one additional feature is based and that are associated with a time prior to a current time. 
     
     
         36 . The method of  claim 31 , wherein updating each partition of data further comprises:
 generating a value for the at least one additional feature associated with a time by applying one or more additional machine learning models to one or more values of one or more characteristics of the content, wherein the one or more additional learning models account for at least one of a decay rate or a propagation delay.   
     
     
         37 . The method of  claim 31 , further comprising:
 generating, using the updated training data set, one or more alternative results; and   replacing the trained machine learning model with a replacement machine learning model, the replacement machine learning model based on the one or more alternative results.   
     
     
         38 . A non-transitory computer-readable storage medium having an executable stored thereon, which when executed, instructs a processor to:
 receive data comprising one or more characteristics of content;   generate a plurality of partitions of the data, wherein each partition of the plurality of partitions of the data:
 comprises values for the one or more characteristics of content, and 
 corresponds to a respective table that identifies the values of the one or more characteristics of the content; 
   updating each partition of data to comprise one or more values of at least one additional feature, wherein the one or more values for at least one additional feature is determined, for each example in a training data set, from values of one or more characteristics of the content on which the additional feature is based and that are included in the partition of data; and   update the training data set to include the one or more values of the additional feature for each example in the training data set to train a machine learning model.   
     
     
         39 . The non-transitory computer-readable storage medium of  claim 38 , wherein the one or more values for at least one additional feature is further determined, for each example in a training data set, from values of one or more characteristics of the content on which the additional feature is based and that are associated with a time prior to a current time. 
     
     
         40 . The non-transitory computer-readable storage medium of  claim 38 , wherein updating each partition of data further comprises:
 generating a value for the at least one additional feature associated with a particular time by applying one or more additional models to one or more values of one or more characteristics of the content, wherein the one or more additional models account for at least one of a decay rate or a propagation delay.