System and method for full results and configuration storage for performance test

Abstract

The present invention enables a performance testing framework that enables multiple components working together to test a deployed application automatically in an unattended manner and to analyze the test results easily. At very high level, the performance testing framework can run performance tests on a tested system with one or more variations without user intervention and save the test results and configuration metadata to a database for later analysis. Each component of the performance testing framework is pluggable and fully configurable. This description is not intended to be a complete description of, or limit the scope of, the invention. Other features, aspects, and objects of the invention can be obtained from a review of the specification, the figures, and the claims.

1 . A system to support performance testing, comprising: 
 a tested system, comprising one or more of: 
 one or more application servers operable to deploy and provide an application to a user; and  
 a load generating tool operable to execute a plurality of test runs to measure performance of the application;  
   an infrastructure, comprising one or more of: 
 an automation controller operable to analyze performance data of the plurality of test runs within and/or between a plurality of variations; and  
   a variation controller operable to: 
 specify the plurality of variations based on properties of the tested system;  
 generate the plurality of test runs for the plurality of variations; and  
 interact with and/or configure the tested system and/or the infrastructure.  
   
     
     
         2 . The system according to  claim 1 , wherein: 
 the application is a Web-based application.    
     
     
         3 . The system according to  claim 1 , wherein: 
 the variation controller is operable to configure the tested system and/or the infrastructure without running the plurality of test runs.    
     
     
         4 . The system according to  claim 1 , wherein: 
 the variation controller is operable to configure one or more of the following: the application to be tested, the one or more servers providing the application, one or more process running on each of one or more servers, and the plurality of test runs.    
     
     
         5 . The system according to  claim 1 , wherein: 
 the variation controller is operable to configure each component of the tested system and/or the infrastructure either automatically or manually with manually-configured setting(s) from the user.    
     
     
         6 . The system according to  claim 1 , wherein: 
 the variation controller is operable to plug or substitute each component of the tested system and/or the infrastructure.    
     
     
         7 . The system according to  claim 1 , wherein: 
 the variation controller is operable to specify configuration data for each component of the tested system and/or the infrastructure.    
     
     
         8 . The system according to  claim 7 , wherein: 
 the configuration data are in the context of and associated with the plurality of test runs.    
     
     
         9 . The system according to  claim 7 , wherein: 
 the variation controller is operable to capture and store one or more of: 
 the configuration data of each component of the tested system and/or the infrastructure;  
 logs of the tested system and/or the infrastructure;  
 scripts of the plurality of test runs; and  
 the performance data of the plurality of test runs.  
   
     
     
         10 . The system according to  claim 9 , wherein: 
 the variation controller is operable to rerun the plurality of test runs under exact the same configuration based on the recorded configuration data.    
     
     
         11 . A method to support performance testing, comprising: 
 specifying a plurality of variations of configuration properties for performance test of a tested system;    executing a plurality of test runs for the plurality of variations;    interacting with and/or configuring the tested system and infrastructure for performance testing; and    analyzing performance data of the plurality of test runs within and/or between the plurality of variations.    
     
     
         12 . The method according to  claim 11 , further comprising: 
 configuring the tested system and the infrastructure without running the plurality of test runs.    
     
     
         13 . The method according to  claim 11 , further comprising: 
 configuring each component of the tested system and/or the infrastructure either automatically or manually with manually-configured setting(s) from the user.    
     
     
         14 . The method according to  claim 11 , further comprising: 
 plugging or substituting each component of the tested system and/or the infrastructure.    
     
     
         15 . The method according to  claim 11 , further comprising: 
 specifying configuration data for each component of the tested system and/or the infrastructure.    
     
     
         16 . The method according to  claim 15 , further comprising: 
 capturing and storing one or more of: 
 the configuration data of each component of the tested system and/or the infrastructure;  
 logs of the tested system and/or the infrastructure;  
 scripts of the plurality of test runs; and  
 the performance data of the plurality of test runs.  
   
     
     
         17 . The method according to  claim 16 , further comprising: 
 rerunning the plurality of test runs under exact the same configuration based on the recorded configuration data.    
     
     
         18 . The method according to  claim 11 , further comprising one or more of: 
 summarizing and reporting analysis result of the performance data to a user;    presenting the analysis result to the user on a Web browser; and    exporting the analysis result to a spreadsheet.    
     
     
         19 . A machine readable medium having instructions stored thereon that when executed cause a system to: 
 specify a plurality of variations of configuration properties for performance test of a tested system;    generate a plurality of test runs for the plurality of variations;    manage to execute each of the plurality of test runs in a plurality of phases for each of the plurality of variations; and    analyze performance data of the plurality of test runs within and/or between the plurality of variations.    
     
     
         20 . A system to support performance testing, comprising: 
 means for specifying a plurality of variations of configuration properties for performance test of a tested system;    means for generating a plurality of test runs for the plurality of variations;    means for managing to execute each of the plurality of test runs in a plurality of phases for each of the plurality of variations; and    means for analyzing performance data of the plurality of test runs within and/or between the plurality of variations.