System and method for performance testing framework

Abstract

The present invention enables a performance testing framework that enables multiple components working together to test a deployed application automatically in an unattended manner and to analyze the test results easily. At very high level, the performance framework can run performance tests on a tested system with one or more variations without user intervention and save the test results and configuration metadata to a database for later analysis. It can also provide reporting facilities to summarize, query, and analyze results information both within a variation and between variations. This description is not intended to be a complete description of, or limit the scope of, the invention. Other features, aspects, and objects of the invention can be obtained from a review of the specification, the FIGS., and the claims.

1 . A system to support performance testing, comprising: 
 a tested system, comprising one or more of: 
 one or more application servers operable to deploy and provide an application to a user;  
 a load generating tool operable to conduct each of a plurality of test runs automatically to measure performance of the application without user intervention; and  
 a performance test database operable to store performance data and/or configuration of the plurality of test runs;  
   an infrastructure, comprising one or more of: 
 a results database operable to store analysis result of the performance data; and  
 an automation controller operable to: 
 extract performance data of the plurality of test runs from the performance test database;  
 analyze the performance data within and/or between the plurality of variations;  
 store analysis result in the results database; and summarize and report the analysis result to the user; and  
 
   a variation controller operable to: 
 interact with the tested system and the infrastructure;  
 configure the tested system automatically based on its configuration properties;  
 specify a plurality of variations of the configuration properties; and  
 generate the plurality of test runs for the load generating tool based on the plurality of variations.  
   
     
     
         2 . The system according to  claim 1 , wherein: 
 the tested system and the infrastructure reside on a plurality of host machines.    
     
     
         3 . The system according to  claim 2 , wherein: 
 the performance test database and the results database reside on different host machines.    
     
     
         4 . The system according to  claim 1 , wherein: 
 the variation controller is implemented via a Java Virtual machine (JVM).    
     
     
         5 . The system according to  claim 1 , wherein the tested system further comprising one or more of: 
 an admin server operable to manage the application; and    one or more proxy servers for the application provided.    
     
     
         6 . The system according to  claim 1 , wherein: 
 the application is a portal of a Web-based service.    
     
     
         7 . The system according to  claim 1 , wherein: 
 performance of the application is measured by one or more of: response time, throughput, capacity, and scalability.    
     
     
         8 . The system according to  claim 1 , wherein: 
 the plurality of variations are different kinds of one or more of: installer, application, portal, network configuration, domain, startup script, database, proxy, one or more configured portal entities, and one or more performance test properties.    
     
     
         9 . The system according to  claim 1 , wherein: 
 each of the plurality of test runs is a ramp-up run or a flat run.    
     
     
         10 . The system according to  claim 1 , wherein: 
 each of the plurality of test runs is a singlerun or a multirun, wherein the multirun is series of dynamically generated test runs varying zero or more configuration properties.    
     
     
         11 . The system according to  claim 10 , wherein: 
 a multirun is divide-and-conquer, goal-based, or incremental-based.    
     
     
         12 . The system according to  claim 1 , wherein: 
 each of the plurality of test runs can be conducted in one or more of the following usage models: run the entire test run automatically, run part of a test run automatically and part of the test run manually, combination of the prior two, re-run executed variations manually, and add configuration properties to extend automation run capability.    
     
     
         13 . The system according to  claim 1 , wherein: 
 each of the plurality of test runs can be launched and run at high, mid, and low level targets.    
     
     
         14 . The system according to  claim 1 , wherein: 
 the performance data of the plurality of test runs can be gathered from one or more of: metadata, context, test run files, server and proxy log files, error log files, and load running reports.    
     
     
         15 . The system according to  claim 1 , wherein: 
 the automation controller is implemented via a state machine.    
     
     
         16 . The system according to  claim 1 , wherein: 
 the automation controller is operable to perform at least one of:    storing the analysis result in a plurality of results directories;    presenting the analysis result to the user on a Web browser; and    export the analysis result to a spreadsheet.    
     
     
         17 . The system according to  claim 1 , wherein: 
 the variation controller is operable to adopt one of more of the following types of communication: 
 invoking a target on a remote host of the one or more application servers and writing ant output to a log file;  
 starting the automation controller and/or the state machine; and using SSH protocol to communicate with the remote host.  
   
     
     
         18 . A method to support performance testing, comprising: 
 specifying a plurality of variations of configuration properties for performance test of a tested system;    configuring the tested system automatically based on the configuration properties;    generating a plurality of test runs based on the plurality of variations;    conducting the plurality of test runs to measure performance of the tested system automatically without user interaction;    storing performance data of the plurality of test runs of the plurality of test runs to a performance test database;    extracting the performance data from the performance test database and analyzing it within and/or between the plurality of variations via a state machine;    storing analysis result of the performance data in a results database; and summarizing and reporting the analysis result to a user.    
     
     
         19 . The method according to  claim 18 , further comprising: 
 running one or more low-level tools manually to cover an area not fully supported by the plurality of test runs.    
     
     
         20 . The method according to  claim 18 , further comprising one or more of: 
 storing configuration of the plurality of test runs to the performance test database for repeat of the plurality of test runs;    measuring performance of the tested system by one or more of: response time, throughput, capacity, and scalability;    launching and running each of the plurality of test runs at high, mid, and low level targets;    storing the analysis result in a plurality of results directories;    presenting the analysis result to the user on a Web browser; and export the analysis result to a spreadsheet.    
     
     
         21 . A machine readable medium having instructions stored thereon that when executed cause a system to: 
 specify a plurality of variations of configuration properties for performance test of a tested system;    configure the tested system automatically based on the configuration properties;    generate a plurality of test runs based on the plurality of variations; conduct the plurality of test runs to measure performance of the tested system automatically without user interaction;    store performance data and/or configuration of the plurality of test runs of the plurality of test runs to a performance test database;    extract the performance data from the performance test database and analyze it within and/or between the plurality of variations via a state machine;    store analysis result of the performance data in a results database; and summarize and report the analysis result to a user.    
     
     
         22 . A system to support performance testing, comprising: 
 means for specifying a plurality of variations of configuration properties for performance test of a tested system;    means for configuring the tested system automatically based on the configuration properties;    means for generating a plurality of test runs based on the plurality of variations;    means for conducting the plurality of test runs to measure performance of the tested system automatically without user interaction;    means for storing performance data and/or configuration of the plurality of test runs of the plurality of test runs to a performance test database;    means for extracting the performance data from the performance test database and analyzing it within and/or between the plurality of variations via a state machine;    means for storing analysis result of the performance data in a results database; and    means for summarizing and reporting the analysis result to a user.