Auto-executing tool for developing test harness files

Abstract

A test harness development component streamlines the design of regression test harnesses for testing text-based software. An auto-recording tool to automatically record the input and output data in an actual run of the testing session and to automatically generate a test harness. The test harness development component provides an auto-executing tool to automatically run the test harness in subsequent regression tests. The test harness language is used to specify the tests, validation rules of the results, and any other rules for running the test of a test harness. The test harness language is human-readable and emulates the manual test procedures with very little overhead.

1. A method, comprising the steps of:
 reading, without user intervention, an automatic regression test of a test harness file; 
 reading non-graphical software to be tested; 
 executing, without user intervention, the automatic regression test and producing test results from the non-graphical software; and 
 determining, without user intervention, whether the test results pass or fail by their comparison to contents of a benchmark file, if the benchmark file is associated with the test harness file; 
 tabulating, without user intervention, pass/fail statistics; 
 recording, without user intervention, the pass/fail statistics; 
 displaying at least a portion of the test results if the benchmark file is associated with the test harness; 
 displaying complete test results to the user if no benchmark file is associated with the test harness; and 
 masking a time-stamp portion of the test results through employment of a subroutine that converts the time-stamp portion of the test results to a pre-determined string prior to comparing the results to the contents of the benchmark file. 
 
     
     
       2. The method of  claim 1  further comprising the step of displaying complete test results of any test of the test harness that fails. 
     
     
       3. The method of  claim 1  further comprising the step of receiving a user command to auto-execute a named test harness file. 
     
     
       4. The method of  claim 1  further comprising the step of automatically capturing the test results. 
     
     
       5. The method of  claim 1  further comprising the step of automatically capturing the test results in response to receiving a user command to capture the test results. 
     
     
       6. The method of  claim 5  further comprising the step of saving the test results in a new benchmark file. 
     
     
       7. The method of  claim 1  wherein the step of filtering comprises the step of discarding a portion of the test results. 
     
     
       8. The method of  claim 1  further comprising the step of filtering the contents of the benchmark file prior to comparing the results. 
     
     
       9. The method of  claim 1  wherein the test harness file comprises a series of named tests, the method further comprising the step of receiving a user-defined test name and wherein the step of determining whether the test results pass or fail is directed only to a test associated with the user-defined test name. 
     
     
       10. The method of  claim 1 , wherein the step of filtering comprises the step of:
 suppressing a portion of the test results that vary between invocations of the non-graphical software to be tested. 
 
     
     
       11. The method of  claim 1 , wherein the subroutine comprises a user defined filter statement within the test harness file. 
     
     
       12. A method, comprising the steps of:
 reading, without user intervention, an automatic regression test of a named test harness file; 
 reading non-graphical software to be tested; 
 executing, without user intervention, the automatic regression test and producing test results from the non-graphical software; 
 determining, without user intervention, whether the test results pass or fail by their comparison to contents of a benchmark file; 
 if a benchmark file is associated with the named test harness file, determining, without user intervention, whether the test results pass or fail by their comparison to contents of the benchmark file associated with the test harness file; 
 if no benchmark file is associated with the named test harness file, capturing the test results and saving the test results to a benchmark file associated with the named test harness file; 
 displaying at least a portion of the test results if a benchmark file is associated with the test harness; 
 displaying complete test results to the user if no benchmark file is associated with the test harness; and 
 masking a time-stamp portion of the test results through employment of a subroutine that converts the time-stamp portion of the test results to a pre-determined string prior to comparing the results to the contents of the benchmark file. 
 
     
     
       13. The method of  claim 12  wherein the method further comprises, if no benchmark file is associated with the named test harness file, the steps of:
 re-executing, without user intervention, the automatic regression test and producing test results from the non-graphical software; 
 determining, without user intervention, whether the test results pass or fail by their comparison to contents of the benchmark file associated with the test harness file; and 
 tabulating, without user intervention, pass/fail statistics. 
 
     
     
       14. The method of  claim 12  further comprising the step of recording, without user intervention, the pass/fail statistics. 
     
     
       15. The method of  claim 12  wherein the step of filtering comprises the step of discarding a portion of the test results. 
     
     
       16. The method of  claim 12  further comprising the step of filtering the contents of the benchmark file prior to comparing the results. 
     
     
       17. The method of  claim 12 , wherein the step of filtering comprises the step of:
 suppressing a portion of the test results that vary between invocations of the non-graphical software to be tested. 
 
     
     
       18. The method of  claim 12 , wherein the subroutine comprises a user defined filter statement within the test harness file. 
     
     
       19. A method, comprising the steps of:
 receiving a user command to generate a benchmark file, using a named test harness file; 
 reading, without user intervention, automatic regression tests of a named test harness file; 
 reading, without user intervention, non-graphical software to be tested; 
 executing, without user intervention, the automatic regression tests and producing test results from the non-graphical software; 
 capturing, without user intervention, the test results; 
 saving, without user intervention, the test results in a benchmark file; and 
 masking a time-stamp portion of the test results through employment of a subroutine that converts the time-stamp portion of the test results to a pre-determined string prior to comparing the results to the contents of the benchmark file. 
 
     
     
       20. The method of  claim 19  further comprising the step of associating the benchmark file with the test harness file. 
     
     
       21. The method of  claim 20  wherein the step of associating comprises naming the benchmark file with at least a portion of the name of the test harness file. 
     
     
       22. The method of  claim 19  further comprising the steps of:
 determining, without user intervention, whether the test results pass or fail by their comparison to contents of a benchmark file; 
 tabulating, without user intervention, pass/fail statistics; and 
 displaying the pass/fail statistics to the user. 
 
     
     
       23. The method of  claim 19  wherein the step of filtering comprises the step of discarding a portion of the test results. 
     
     
       24. The method of  claim 23  further comprising the step of filtering the contents of the benchmark file prior to comparing the results.