System and method for error pattern matching for performance test

Abstract

The present invention enables a performance testing framework that enables multiple components working together to test a deployed application automatically in an unattended manner and to analyze the test results easily. At very high level, the performance testing framework can run performance tests on a tested system with one or more variations without user intervention and save the test results and configuration metadata to a database for later analysis. In addition, the framework can monitor, detect and report issues in the performance tests via pattern matching. This description is not intended to be a complete description of, or limit the scope of, the invention. Other features, aspects, and objects of the invention can be obtained from a review of the specification, the figures, and the claims.

1 . A system to support performance testing, comprising: 
 a tested system, comprising one or more of: 
 one or more application servers operable to deploy and provide an application to a user; and  
 a load generating tool operable to execute a plurality of test runs to measure performance of the application automatically;  
   an infrastructure, comprising one or more of: 
 an automation controller operable to analyze performance data of the plurality of test runs within and/or between a plurality of variations; and  
   a variation controller operable to: 
 generating the plurality of test runs based on a plurality of variations of configuration properties of the tested system;  
 monitoring the execution of the plurality of test runs; and  
 detecting an issue by scanning and pattern matching a plurality of files of the plurality of test runs.  
   
     
     
         2 . The system according to  claim 1 , wherein: 
 the application is a Web-based application.    
     
     
         3 . The system according to  claim 1 , wherein: 
 each of the plurality of files is one of: a log file, a database creation output file, and a command output file for problems.    
     
     
         4 . The system according to  claim 1 , wherein: 
 the issue is classified by severity.    
     
     
         5 . The system according to  claim 1 , wherein: 
 the variation controller is further operable to report and/or save context information of the issue when it is detected.    
     
     
         6 . The system according to  claim 1 , wherein: 
 the variation controller is operable to stop one of the plurality of test runs and advance to the next one when the issue is detected.    
     
     
         7 . The system according to  claim 1 , wherein: 
 the variation controller is operable to utilize one or more pattern files containing a plurality of patterns for pattern matching.    
     
     
         8 . The system according to  claim 7 , wherein: 
 each of the plurality of patterns is specified using Java Regular Expressions Pattern regular expression format.    
     
     
         9 . The system according to  claim 7 , wherein: 
 the variation controller is operable to detect when the issue occurs based on types of the plurality of patterns.    
     
     
         10 . A method to support performance testing, comprising: 
 generating and executing a plurality of test runs based on a plurality of variations of configuration properties of a tested system;    analyzing performance data of the plurality of test runs within and/or between the plurality of variations;    monitoring the execution of the plurality of test runs; and    detecting an issue by scanning and pattern matching a plurality of files of the plurality of test runs.    
     
     
         11 . The method according to  claim 10 , further comprising: 
 classifying the issue by severity.    
     
     
         12 . The method according to  claim 10 , further comprising: 
 reporting and/or saving context information of the issue when it is detected.    
     
     
         13 . The method according to  claim 10 , further comprising: 
 stopping one of the plurality of test runs and advancing to the next one when the issue is detected.    
     
     
         14 . The method according to  claim 10 , further comprising: 
 utilizing one or more pattern files containing a plurality of patterns for pattern matching.    
     
     
         15 . The method according to  claim 15 , further comprising one or more of: 
 specifying each of the plurality of patterns is using Java Regular Expressions Pattern regular expression format; and    detecting when the issue occurs based on types of the plurality of patterns.    
     
     
         16 . The method according to  claim 10 , further comprising one or more of: 
 summarizing and reporting analysis result of the performance data to a user;    storing the performance data and/or the analysis result to a database;    presenting the analysis result to the user on a Web browser; and    exporting the analysis result to a spreadsheet.    
     
     
         17 . A machine readable medium having instructions stored thereon that when executed cause a system to: 
 generate and execute a plurality of test runs based on a plurality of variations of configuration properties of a tested system;    analyze performance data of the plurality of test runs within and/or between the plurality of variations;    monitor the execution of the plurality of test runs; and    detect an issue by scanning and pattern matching a plurality of files of the plurality of test runs.    
     
     
         18 . A system to support performance testing, comprising: 
 means for generating and executing a plurality of test runs based on a plurality of variations of configuration properties of a tested system;    means for analyzing performance data of the plurality of test runs within and/or between the plurality of variations;    means for monitoring the execution of the plurality of test runs; and    means for detecting an issue by scanning and pattern matching a plurality of files of the plurality of test runs.