Automated test case result analyzer

Abstract

A test result analyzer for processing results of testing software. The analyzer has an interface emulating the interface of a traditional data logger. After analyzing the test results, selected results may be output to a log file or otherwise reported for subsequent use. The test result analyzer compares test results to results in a database of historical data from running test cases. The analyzer filters out results representative of fault conditions already reflected in the historical data, thereby reducing the amount of data that must be processed to identify fault conditions.

1 . A method of testing software, comprising the acts: 
 a) providing a plurality of records, each record comprising failure symptom data of a fault condition associated with the software;    b) automatically comparing failure symptom data derived from subjecting the software to a test case to the failure symptom data of one or more of the plurality of records; and    c) selectively reporting a test result based on the comparison in the act b).    
     
     
         2 . The method of  claim 1 , wherein the act a) comprises providing each record in a portion of the plurality of records with a fault signature associated with a failure of the software when subjected to a test case.  
     
     
         3 . The method of  claim 2 , wherein the act a) comprises providing each record in a second portion of the plurality of records with a fault signature associated with a mis-configuration of the test environment.  
     
     
         4 . The method of  claim 2 , wherein the act c) comprises reporting the test result when the failure symptom data derived from subjecting the software to the test case does not match failure symptom data stored in any of the plurality of records.  
     
     
         5 . The method of  claim 1 , wherein the act a) comprises adding records to the plurality of records as failures occur during testing of the software.  
     
     
         6 . The method of  claim 1 , additionally comprising the act: 
 d) reporting to a human user statistics of test results having failure symptom data that matches failure symptom data in one of the plurality of records.    
     
     
         7 . The method of  claim 6 , wherein the act c) comprises selectively writing a record of the test result in a log file.  
     
     
         8 . The method of  claim 1 , wherein the failure symptom data in each of the plurlaity of records comprises a stack trace and the act b) comprises comparing a stack trace derived from subjecting the software to a test case to the stack trace of one or more of the plurality of records.  
     
     
         9 . A computer-readable medium having computer-executable components comprising: 
 a) a component for storing historical failure information;    b) a component for receiving a plurality of test results;    c) a component for filtering the plurality of test results to provide filtered test results representing failures not in the historical failure information; and    d) a component for reporting the filtered test results.    
     
     
         10 . The computer-readable medium of  claim 9 , wherein the component for receiving a test result comprises a logging interface of a test harness.  
     
     
         11 . The computer-readable medium of  claim 1 , wherein the component for filtering comprises an analysis engine applying a plurality of rules specifying conditions under which a test result of the plurality of test results is deemed to be in the historical failure information.  
     
     
         12 . The computer-readable medium of  claim 11 , wherein the plurality of rules comprises default rules and user supplied rules.  
     
     
         13 . The computer-readable medium of  claim 9 , additionally comprising a component for analyzing the plurality of test results to identify a generic problem.  
     
     
         14 . The computer-readable medium of  claim 13 , wherein the component for analyzing the plurality of test results to identify a generic problem detects a mis-configuration of the test system.  
     
     
         15 . The computer-readable medium of  claim 9 , wherein the components a), b), c), and d) are each implemented as a class library.  
     
     
         16 . A method of testing software, comprising the acts: 
 a) providing a plurality of records, each record comprising failure symptom data associated with a previously identified fault condition;    b) obtaining a plurality of test results, with at least a portion of the plurality of test results indicating a failure condition and having failure symptom data associated therewith; and    c) automatically filtering the plurality of test results to produce a filtered result comprising selected ones of the plurality of test results having failure symptom data that represents a failure condition not reflected in the plurality of records.    
     
     
         17 . The method of  claim 16 , wherein the act b) of obtaining a plurality of test results comprises applying a plurality of test cases to the software.  
     
     
         18 . The method of  claim 16 , wherein the act a) of providing a plurality of records comprises converting a failure log in a specific format to a generic format.  
     
     
         19 . The method of  claim 16 , additionally comprising the act d) of recording the filtered result.  
     
     
         20 . The method of  claim 19 , wherein the act d) comprises writing the filtered result to an XML file.