Page simulation system

Abstract

The disclosed computer-implemented method may include accessing updated data structures that are to be included in a user interface functionality test, where the updated data structures contribute to a user interface. The method may also include accessing live or snapshotted data captured from services running in a production environment, initiating generation of a first user interface instance using the updated data structures and using the accessed live or snapshotted data, and initiating generation of a second user interface instance using a different version of the data structures and using the same accessed live or snapshotted data. The method further includes comparing the first user interface instance to the second user interface instance to identify differences and then determine which outcome-defining effects the updated data structures had on the user interface based on the identified differences between the user interfaces. Various other methods, systems, and computer-readable media are also disclosed.

What is claimed is: 
     
       1. A computer-implemented method comprising:
 accessing, by a controller, one or more updated data structures that are to be included in a user interface functionality test, the updated data structures contributing at least partially to a user interface; 
 accessing, by the controller, a portion of live or snapshotted data captured from one or more services running in a production environment, the services being initialized at a specified common time identified in a shared contract that is shared among the services, the shared contract providing a common notion of time among the services, wherein the live or snapshotted data is used in the user interface functionality test, and wherein the live or snapshotted data includes service data generated by the services and user input data identifying user inputs received at the user interface, wherein the services implement queue-based communication that decouples the services and provides individual request retries; 
 initiating, by the controller, generation of a first simulated user interface instance using the updated data structures and using the accessed live or snapshotted data, the first simulated user interface portraying how the updated data structures would function if exposed to external users when hosted by the production environment; 
 initiating, by the controller, generation of a second simulated user interface instance using a different version of the updated data structures and using the accessed live or snapshotted data, the second simulated user interface portraying how the different version of the updated data structures would function if exposed to external users when hosted by the production environment, wherein the first and second simulated user interface instances are generated within the production environment but are inaccessible to external users; 
 computing one or more metrics related to the first and second user interface instances, wherein the computation is performed in a separate computing stage, the separate computing stage allowing the metrics to be changed during computation without regenerating at least a portion of the first and second user interface instances; 
 comparing, by the controller, the first user interface instance to the second user interface instance to identify one or more differences between the first user interface instance and the second user interface instance according to the live or snapshotted data and according to the user input data identifying user inputs received at the user interface; and 
 based on the comparison, determining, by the controller, one or more outcome-defining effects the updated data structures had on the user interface based on the identified differences between the first user interface instance and the second user interface instance. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein the controller further coordinates the one or more services to access the live or snapshotted data starting at a specified point in time. 
     
     
       3. The computer-implemented method of  claim 2 , wherein the controller further coordinates the one or more services to access a common clock based on the specified point in time. 
     
     
       4. The computer-implemented method of  claim 2 , wherein the controller establishes a contract between the one or more services to ensure that the one or more services use a same version of metadata and to ensure that the services access a same live or snapshotted data. 
     
     
       5. The computer-implemented method of  claim 1 , wherein the snapshotted data includes inputs received at the one or more services in addition to outputs generated by the one or more services. 
     
     
       6. The computer-implemented method of  claim 1 , wherein the one or more services comprise stateful services running in the production environment, at least a portion of state information being stored for each service. 
     
     
       7. The computer-implemented method of  claim 1 , wherein the one or more updated data structures comprise at least one of software code, configuration files, or machine learning models. 
     
     
       8. The computer-implemented method of  claim 1 , wherein the controller modifies the one or more services to generate one or more modified versions of the services, such that calls for data between the services are routed to the modified versions of the services. 
     
     
       9. The computer-implemented method of  claim 8 , wherein the one or more modified versions of the services are instantiated to process a batch of tasks and are automatically shut down upon completion of the batch of tasks. 
     
     
       10. The computer-implemented method of  claim 9 , wherein the modified versions of the services are instantiated to process the batch of tasks are assigned a specified data snapshot to use when processing the batch of tasks. 
     
     
       11. The computer-implemented method of  claim 1 , wherein the controller switches from live data to snapshotted data or switches from snapshotted data to live data upon receiving an indication to make a switch. 
     
     
       12. A system comprising:
 at least one physical processor; 
 physical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to:
 access, by a controller, one or more updated data structures that are to be included in a user interface functionality test, the updated data structures contributing at least partially to a user interface; 
 access, by the controller, a portion of live or snapshotted data captured from one or more services running in a production environment, the services being initialized at a specified common time identified in a shared contract that is shared among the services, the shared contract providing a common notion of time among the services, wherein the live or snapshotted data is used in the user interface functionality test, and wherein the live or snapshotted data includes service data generated by the services and user input data identifying user inputs received at the user interface, wherein the services implement queue-based communication that decouples the services and provides individual request retries; 
 initiate, by the controller, generation of a first simulated user interface instance using the updated data structures and using the accessed live or snapshotted data, the first simulated user interface portraying how the updated data structures would function if exposed to external users when hosted by the production environment; 
 initiate, by the controller, generation of a second simulated user interface instance using a different version of the updated data structures and using the accessed live or snapshotted data, the second simulated user interface portraying how the different version of the updated data structures would function if exposed to external users when hosted by the production environment, wherein the first and second simulated user interface instances are generated within the production environment but are inaccessible to external users; 
 computing one or more metrics related to the first and second user interface instances, wherein the computation is performed in a separate computing stage, the separate computing stage allowing the metrics to be changed during computation without regenerating at least a portion of the first and second user interface instances; 
 compare, by the controller, the first user interface instance to the second user interface instance to identify one or more differences between the first user interface instance and the second user interface instance according to the live or snapshotted data and according to the user input data identifying user inputs received at the user interface; and 
 based on the comparison, determine, by the controller, one or more outcome-defining effects the updated data structures had on the user interface based on the identified differences between the first user interface instance and the second user interface instance. 
 
 
     
     
       13. The system of  claim 12 , further comprising computing one or more metrics to establish a quality level of simulated behavior indicated by the outcome-defining effects. 
     
     
       14. The system of  claim 13 , wherein computing the one or more metrics includes altering one or more user interface objects shown in the user interface to remove bias in the one or more outcome-defining effects. 
     
     
       15. The system of  claim 13 , wherein the metrics give at least partial credit for similarities in outcome-defining effects identified in the comparison between the first user interface instance and the second user interface instance. 
     
     
       16. The system of  claim 12 , wherein comparing the first user interface instance to the second user interface instance comprises comparing at least one of the first user interface instance or the second user interface instance to previous A/B test results. 
     
     
       17. The system of  claim 16 , wherein the comparison to the previous A/B test results indicate whether a new A/B test is to be run. 
     
     
       18. The system of  claim 12 , wherein the second user interface instance is generated using snapshotted data that indicates user inputs over a specified period of time relative to a version of the user interface that was presented to a user at the time a snapshot was taken. 
     
     
       19. The system of  claim 18 , wherein comparing the first user interface instance to the second user interface instance to identify one or more differences between the first user interface instance and the second user interface instance includes comparing the first user interface instance generated using the updated data structures to the second user interface instance generated using the snapshotted data with the indicated user inputs over the specified period of time, such that the comparison of the first user interface instance generated using the updated data structures to the second user interface instance generated using the snapshotted data with the indicated user inputs over the specified time period indicates how the first user interface instance with the one or more updated data structures compares to the second user interface instance using the same snapshotted user inputs. 
     
     
       20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
 access, by a controller, one or more updated data structures that are to be included in a user interface functionality test, the updated data structures contributing at least partially to a user interface; 
 access, by the controller, a portion of live or snapshotted data captured from one or more services running in a production environment, the services being initialized at a specified common time identified in a shared contract that is shared among the services, the shared contract providing a common notion of time among the services, wherein the live or snapshotted data is used in the user interface functionality test, and wherein the live or snapshotted data includes service data generated by the services and user input data identifying user inputs received at the user interface, wherein the services implement queue-based communication that decouples the services and provides individual request retries; 
 initiate, by the controller, generation of a first simulated user interface instance using the updated data structures and using the accessed bye or snapshotted data, the first simulated user interface portraying how the updated data structures would function if exposed to external users when hosted by the production environment; 
 initiate, by the controller, generation of a second simulated user interface instance using a different version of the updated data structures and using the accessed live or snapshotted data, the second simulated user interface portraying how the different version of the updated data structures would function if exposed to external users when hosted by the production environment, wherein the first and second simulated user interface instances are generated within the production environment but are inaccessible to external users; 
 compute one or more metrics related to the first and second user interface instances, wherein the computation is performed in a separate computing stage, the separate computing stage allowing the metrics to be changed during computation without regenerating at least a portion of the first and second user interface instances; 
 compare, by the controller, the first user interface instance to the second user interface instance to identify one or more differences between the first user interface instance and the second user interface instance according to the live or snapshotted data and according to the user input data identifying user inputs received at the user interface; and 
 based on the comparison, determine, by the controller, one or more outcome-defining, effects the updated data structures bad on the user interface based on the identified differences between the first user interface instance and the second user interface instance.