System and method for flexible performance testing

Abstract

The present invention enables a performance testing framework that enables multiple components working together to test a deployed application automatically in an unattended manner and to analyze the test results easily. At very high level, the performance testing framework can run performance tests on a tested system with one or more variations without user intervention and save the test results and configuration metadata to a database for later analysis. The test run for each variation can be run flexibly in a plurality of phases, each of which can be added, configured or skipped for the variation. This description is not intended to be a complete description of, or limit the scope of, the invention. Other features, aspects, and objects of the invention can be obtained from a review of the specification, the figures, and the claims.

1 . A system to support performance testing, comprising: 
 a tested system, comprising one or more of: 
 one or more application servers operable to deploy and provide an application to a user; and  
 a load generating tool operable to execute a plurality of test runs to measure performance of the application;  
   an infrastructure, comprising one or more of: 
 an automation controller operable to analyze performance data of the plurality of test runs within and/or between a plurality of variations;  
   a variation controller operable to: 
 interact with the tested system and the infrastructure;  
 specify the plurality of variations of configuration properties of the tested system;  
 generate the plurality of test runs for the plurality of variations;  
 invoking a dispatcher to process a single run of each of the plurality of test runs; and  
 said dispatcher operable to manage executing each of the plurality of test runs in a plurality of phases for each of the plurality of variations.  
   
     
     
         2 . The system according to  claim 1 , wherein: 
 the application is a Web-based application.    
     
     
         3 . The system according to  claim 1 , wherein: 
 each of the plurality of variations specifies one or more of the following per-phase component configuration settings: installer, application, portal, network configuration, domain, startup script, database, proxy, one or more configured portal entities, and one or more performance test properties.    
     
     
         4 . The system according to  claim 1 , wherein: 
 the performance data of the application is measured by one or more of: response time, throughput, capacity, and scalability.    
     
     
         5 . The system according to  claim 1 , wherein: 
 the dispatcher is operable to compute per-phase configuration settings of the plurality of phases and execute the plurality of phases with those settings when executing one of the plurality of test runs automatically.    
     
     
         6 . The system according to  claim 5 , wherein: 
 the dispatcher is operable to skip one or more of the plurality of phases during execution of each of the plurality of test runs based on the per-phase configuration settings and history.    
     
     
         7 . The system according to  claim 1 , wherein: 
 the dispatcher is operable to monitor, detect, and report the performance data and/or issues during execution of the plurality of phases of one of the plurality of test runs.    
     
     
         8 . The system according to  claim 7 , wherein: 
 the dispatcher is operable to skip one or more of the plurality of phases based on the performance data and/or issues of one or more of its proceeding phases.    
     
     
         9 . The system according to  claim 7 , wherein: 
 the dispatcher is operable to organize and store the performance result and/or issues into a plurality of results directories.    
     
     
         10 . The system according to  claim 7 , wherein: 
 the dispatcher is operable to end the current test run based on performance data and/or issues of the current test run, variation, or the plurality of phases.    
     
     
         11 . A method to support performance testing, comprising: 
 specifying a plurality of variations of configuration properties for performance test of a tested system;    generating a plurality of test runs for the plurality of variations;    managing to execute each of the plurality of test runs in a plurality of phases for each of the plurality of variations; and    analyzing performance data of the plurality of test runs within and/or between the plurality of variations.    
     
     
         12 . The method according to  claim 11 , further comprising: 
 measuring performance of the tested system by one or more of: response time, throughput, capacity, and scalability.    
     
     
         13 . The method according to  claim 11 , further comprising: 
 computing per-phase configuration settings of the plurality of phases and executing the plurality of phases with those settings when executing one of the plurality of test runs automatically.    
     
     
         14 . The method according to  claim 13 , further comprising: 
 skipping one or more of the plurality of phases during execution of each of the plurality of test runs based on the per-phase configuration settings and history.    
     
     
         15 . The method according to  claim 11 , further comprising: 
 monitoring, detecting, and reporting the performance data and/or issues during execution of the plurality of phases each of the plurality of test runs.    
     
     
         16 . The method according to  claim 15 , further comprising one or more of: 
 skipping one or more of the plurality of phases based on the performance data and/or issues of one or more of its proceeding phases;    organizing and storing the performance result and/or issues into a plurality of results directories; and    ending the current test run based on performance data and/or issues of the current test run, variation, or the plurality of phases.    
     
     
         17 . The method according to  claim 11 , further comprising one or more of: 
 storing the performance data and/or configuration of the plurality of test runs to a performance test database;    summarizing and reporting analysis result of the performance data to a user;    presenting the analysis result to the user on a Web browser; and    exporting the analysis result to a spreadsheet.    
     
     
         18 . A machine readable medium having instructions stored thereon that when executed cause a system to: 
 specify a plurality of variations of configuration properties for performance test of a tested system;    generate a plurality of test runs for the plurality of variations;    manage to execute each of the plurality of test runs in a plurality of phases for each of the plurality of variations; and    analyze performance data of the plurality of test runs within and/or between the plurality of variations.    
     
     
         19 . A system to support performance testing, comprising: 
 means for specifying a plurality of variations of configuration properties for performance test of a tested system;    means for generating a plurality of test runs for the plurality of variations;    means for managing to execute each of the plurality of test runs in a plurality of phases for each of the plurality of variations; and    means for analyzing performance data of the plurality of test runs within and/or between the plurality of variations.