Testing of application service versions on live data

Abstract

Some embodiments include a computer server implementing a splitter engine to perform testing of a pre-production version of an application service against live traffic. The splitter engine can receive a client request for a live application service. The splitter engine can generate a live response by processing the client request through a live instance of a production engine. The splitter engine can select one or more request processors to process the client request and a processing order based on a request type of the client request. The splitter engine can process the client request through the one or more request processors according to the processing order after responding to the client request with the live response. The splitter engine can compare states of at least two of the production engine and the one or more request processors.

What is claimed is: 
     
       1. A computer-implemented method, comprising:
 receiving, from a client, a client request for an application service; 
 responding to the client request with a live response by causing the client request to be processed by a production engine to generate the live response; 
 determining a first state of the production engine as a result of processing the client request to generate the live response; 
 identifying, based on the received client request, one or more request processors to process the client request and a processing order, wherein the one or more request processors includes a shadow engine, which is a potential production engine being tested against live traffic without interfering with the production engine; 
 causing the received client request to be processed via the shadow engine after responding to the client request with the live response; 
 determining a second state of the shadow engine as a result of processing the client request; and 
 identifying a difference between the states of the shadow engine and the production engine. 
 
     
     
       2. The computer-implemented method of  claim 1 , further comprising storing the difference between the states of the shadow engine and the production engine as an error entry in an anomaly report; and wherein the error entry includes the client request that caused the difference between the states of the shadow engine and the production engine. 
     
     
       3. The computer-implemented method of  claim 1 , wherein determining the one or more request processors includes:
 determining that the client request requires no modification to a production database maintained by the production engine; and 
 responsive to determining that the client request requires no modification, scheduling to process the client request through the shadow engine in response to responding to the client request with the live response. 
 
     
     
       4. The computer-implemented method of  claim 1 , further comprising:
 processing the client request through a third engine after processing the client request through the shadow engine; and 
 comparing states of the third engine and the production engine. 
 
     
     
       5. The computer-implemented method of  claim 1 , wherein processing the client request includes generating a test response via the shadow engine; and wherein identifying the difference includes comparing the live response and the test response. 
     
     
       6. A computer-readable storage memory apparatus storing computer-executable instructions, comprising:
 instructions for receiving a client request for a live application service, wherein the client request indicates a dataset modification operation; 
 instructions for causing the client request to be processed via a live instance of a production engine by modifying a live production database; 
 instructions for causing the client request to be processed via a mirror instance of the production engine to modify a jailed production database after responding to the client request with a live response; 
 instructions for determining a first state of the mirror instance of the production engine as a result of processing the client request; 
 instructions for causing the client request to be processed via a first shadow engine to modify a first shadow database after processing the client request via the mirror instance of the production engine; 
 instructions for determining a second state of the first shadow engine as a result of processing the client request; and 
 instructions for identifying a difference between the states of the mirror instance of the production engine and the first shadow engine. 
 
     
     
       7. The computer-readable storage memory apparatus of  claim 6 , further comprising:
 instructions for causing the client request to be processed via a second shadow engine to modify a second shadow database after processing the client request via the first shadow engine; and 
 instructions for identifying a difference between states of the mirror instance of the production engine and the second shadow engine. 
 
     
     
       8. The computer-readable storage memory apparatus of  claim 6 , further comprising:
 instructions for seeding the jailed production database with a mirror copy of the live production database prior to receiving the client request. 
 
     
     
       9. The computer-readable storage memory apparatus of  claim 8 , further comprising:
 instructions for seeding the shadow database in response to seeding the jailed production database; and 
 wherein the shadow database is seeded with same data content as the jailed production database. 
 
     
     
       10. The computer-readable storage memory apparatus of  claim 6 , further comprising:
 instructions for storing the difference between the states of the mirror instance of the production engine and the first shadow engine as an error entry in an anomaly report 
 instructions for counting a number of error entries in the anomaly report; and 
 instructions for shutting down the first shadow engine in response to the number of error entries reaching a threshold. 
 
     
     
       11. The computer-readable storage memory apparatus of  claim 10 , further comprising:
 instructions for, in response to shutting down the first shadow engine, re-deploying a new instance of the first shadow engine and creating a new anomaly report. 
 
     
     
       12. The computer-readable storage memory apparatus of  claim 6 , wherein the difference between the states of the mirror instance of the production engine and the first shadow engine is a difference in operating performance between a first computing environment executing the first shadow engine and a second computing environment executing the mirror instance of the production engine. 
     
     
       13. The computer-readable storage memory apparatus of  claim 6 , wherein the difference between the states of the mirror instance of the production engine and the first shadow engine is a difference in resource consumption between the first shadow engine and the mirror instance of the production engine. 
     
     
       14. The computer-readable storage memory apparatus of  claim 6 , wherein the difference between the states of the mirror instance of the production engine and the first shadow engine is a difference in operating memory states of the first shadow engine and the mirror instance of the production engine. 
     
     
       15. The computer-readable storage memory apparatus of  claim 6 , wherein the difference between the states of the mirror instance of the production engine and the first shadow engine is a difference in corresponding database entries of the shadow database and the jailed production database respectively. 
     
     
       16. The computer-readable storage memory apparatus of  claim 6 , wherein the difference between the states of the mirror instance of the production engine and the first shadow engine is a difference in request responses respectively produced by the mirror instance of the production engine and the first shadow engine. 
     
     
       17. A computing device, comprising:
 a processor; 
 a network interface component configured to receive a client request for a live application service; 
 a memory for maintaining a test pipeline setting identifying one or more processor destinations for one or more request types; 
 a splitter engine executed via the processor; wherein the splitter engine is configured to:
 generate a live response by causing the client request to be processed through a live instance of a production engine; 
 determine a first state of the production engine as a result of processing the client request to generate the live response; 
 select one or more request processors to process the client request and a processing order based on the test pipeline setting and a request type of the client request; and 
 cause the client request to be processed through the one or more request processors according to the processing order after responding to the client request with the live response; 
 determine a second state of the one or more request processors as a result of processing the client request; and 
 
 a test analytic engine executed via the processor; wherein the test analytic engine is configured to compare the states of at least two of the production engine and the one or more request processors. 
 
     
     
       18. The computing device of  claim 17 , wherein the memory stores one or more patterns indicating an anomaly type that does not amount to an error; and wherein the test analytic engine is configured to store a difference, from the comparing of the states, in an anomaly report when the test analytic engine identifies the difference as matching none of the one or more patterns. 
     
     
       19. The computing device of  claim 18 , wherein the test analytic engine is configured to convert the client request into server-side replayable format and store the converted client request. 
     
     
       20. The computing device of  claim 17 , wherein the one or more request processors includes a shadow engine, a mirror instance of the production engine operating a jailed production database, another splitter engine, or any combination thereof.