Methods and systems for predictive engine evaluation and replay of engine performance

Abstract

Disclosed are methods and systems of tracking the deployment of a predictive engine for machine learning, including steps to deploy an engine variant of the predictive engine based on an engine parameter set, wherein the engine parameter set identifies at least one data source and at least one algorithm; receive one or more queries to the deployed engine variant from one or more end-user devices, and in response, generate predicted results; receive one or more actual results corresponding to the predicted results; associate the queries, the predicted results, and the actual results with a replay tag, and record them with the corresponding deployed engine variant.

What is claimed is: 
     
         1 . A system for tracking a predictive engine for replay of engine performance, comprising:
 a processor;   an engine variant of the predictive engine stored in a digital working memory, wherein the engine variant is determined by an engine parameter set, and wherein the engine parameter set identifies at least one data source and at least one algorithm; and   a non-transitory, computer-readable storage medium for storing program code, the program code when executed by the processor, causes the processor to:
 deploy the engine variant of the predictive engine based on the engine parameter set; 
 receive one or more queries to the deployed engine variant from one or more end-user devices; 
 in response to the queries, the deployed engine variant generates one or more predicted results; 
 receive one or more actual results corresponding to the predicted results; and 
 associate the queries, the predicted results, and the actual results with a replay tag, and record the queries, the predicted results, and the actual results with the corresponding deployed engine variant. 
   
     
     
         2 . The system of  claim 1 , wherein the program code when executed by the processor, further causes the processor to:
 receive a replay request specified by one or more replay tags; and   in response to the replay request, replay at least one item selected from the group consisting of the queries, the predicted results, and the actual results associated with the one or more replay tags.   
     
     
         3 . The system of  claim 1 , wherein the engine parameter set is generated manually by an operator. 
     
     
         4 . The system of  claim 1 , wherein the engine parameter set is determined automatically. 
     
     
         5 . The system of  claim 1 , wherein the actual results comprise a sequence of user responses. 
     
     
         6 . The system of  claim 1 , wherein the actual results comprise a sequence of user responses collected over a delayed time frame. 
     
     
         7 . The system of  claim 1 , wherein the actual results comprise a sequence of user responses recorded from at least one cohort of users. 
     
     
         8 . The system of  claim 1 , wherein the actual results are received from a datastore. 
     
     
         9 . The system of  claim 1 , wherein the actual results are simulated. 
     
     
         10 . A method of tracking a predictive engine for replay of engine performance, comprising:
 deploying an engine variant of the predictive engine based on an engine parameter set, wherein the engine parameter set identifies at least one data source and at least one algorithm;   receiving one or more queries from one or more end-user devices;   in response to the queries, the deployed engine variant generating one or more predicted results;   receiving one or more actual results corresponding to the predicted results; and   associating the queries, the predicted results, and the actual results with a replay tag, and recording the queries, the predicted results, and the actual results with the corresponding deployed engine variant;   
     
     
         11 . The method of  claim 10 , further comprising:
 receiving a replay request specified by one or more replay tags; and   in response to the replay request, replaying at least one item selected from the group consisting of the queries, the predicted results, and the actual results associated with the one or more replay tags.   
     
     
         12 . The method of  claim 10 , wherein the engine parameter set is generated manually by an operator. 
     
     
         13 . The method of  claim 10 , wherein the engine parameter set is determined automatically. 
     
     
         14 . The method of  claim 10 , wherein the actual results comprise a sequence of user responses. 
     
     
         15 . The method of  claim 10 , wherein the actual results comprise a sequence of user responses collected over a delayed time frame. 
     
     
         16 . The method of  claim 10 , wherein the actual results comprise a sequence of user responses recorded from at least one cohort of users. 
     
     
         17 . The method of  claim 10 , wherein the actual results are received from a datastore. 
     
     
         18 . The method of  claim 10 , wherein the actual results are simulated. 
     
     
         19 . A non-transitory computer-readable storage medium for tracking a predictive engine for replay of engine performance, the storage medium comprising program code stored thereon, that when executed by a processor, causes the processor to:
 deploy an engine variant of the predictive engine based on an engine parameter set, wherein the engine parameter set identifies at least one data source and at least one algorithm;   receive one or more queries from one or more end-user devices;   in response to the queries, the deployed engine variant generates one or more predicted results;   receive one or more actual results corresponding to the predicted results; and   associate the queries, the predicted results, and the actual results with a replay tag, and record the queries, the predicted results, and the actual results with the corresponding deployed engine variant.   
     
     
         20 . The non-transitory computer-readable storage medium of  claim 19 , wherein the program code when executed by the processor, further causes the processor to:
 receive a replay request specified by one or more replay tags; and   in response to the replay request, replay at least one item selected from the group consisting of the queries, the predicted results, and the actual results associated with the one or more replay tags.