User extensible testing system

Abstract

A user extensible, automated testing system comprises a plurality of modules, each of which is user extensible. The journal module provides the ability to record events representative of user actions taken to execute certain tasks which farm the test. The filter module provides the means to filter out data which is not determinative of the pass or fail status of a test. The save data module provides the means to save data which subsequently compared to master data to determine the pass/fail status of the test. The compare module provides the means to compare the data saved with the master data to determine differences between the data and therefore the pass/fail status of the test. The review module provides a graphical review of the differences between the master data and actual data. Each module is user extensible whereby the user may update, change or delete portions of the module functionality in order to tailor the testing system to a particular computer system or computer software application.

We claim: 
     
       1. In a window-based computer system comprising a central processing unit (CPU), memory, input/output devices comprising a keyboard, mouse and display, wherein the input and output of information is performed through windows defined on the display, a user extensible, automated testing system generating test results for the testing of application programs which compares master data to actual data generated during the testing of the application program, said test results comprising the differences between the master data and actual data, said user extensible, automated testing system comprising: journaling means coupled to said window system for generating and replaying a journal file comprising a list of events representative of user actions performed during a user session which, when replayed, cause the application program to be executed in a predetermined manner which tests the application program;   filter means for eliminating portions of data which are not determinative of the test results, said filter means accepting data generated from the testing of the application programs, said filter means comprising; filter functions which, when executed, filter portions of data, each filter function identified by name;   a filter list file, comprising a list of names of at least one filter function to filter portions of data, said filter list file referenced to determine available filter functions;   filter execution means coupled to the filter list file and the central processing unit such that the filter functions are executed by the central processing unit to filter portions of data according to the filter functions listed in the filter list file; and   filter modification means coupled to the filter list file, filter functions and to the input/output devices through the window system, the filter modification means modifying the filter list file such that filter functions can be added, deleted or changed;     save data means to save data generated during the testing of the application program, said save data means being coupled to the window system and to the filter means such that data output from said filter means is input to said save data means, said save data means comprising; save data functions, which, when executed, save predetermined portions of the data, each save data function identified by a name;   save data execution means coupled to said save data functions and the central processing unit such that said save data functions are executed by the central processing unit to save predetermined portions of the data according to specified save data functions, said save data execution means enabled by a plurality of events placed in the journal file representative of user actions which selects the save data function; and   save data modification means coupled to the save data functions and to the input/output devices through the window system so as to modify the save data functions such that save data functions can be added, deleted or changed;     comparison means for comparing predetermined portions of data from the master data to corresponding portions of data from the actual data, said comparison means being coupled to the save data means such that portions of data saved by the save data functions are input to the comparison means, said comparison means comprising; comparison functions, each identified by name, which, when executed, compare predetermined portions of the master data with corresponding predetermined portions of the actual data, determine the differences between the predetermined portions of the master data and actual data, and cause a failure file to be generated containing the actual data saved which have differences as determined by the comparison function;   comparison execution means coupled to the comparison functions and the central processing unit so as to compare predetermined portions of the master data and actual data according to the comparison functions, said comparison execution means enabled after execution of save data is completed;   comparison modification means to modify the comparison functions, said comparison modification means coupling the comparison functions to the input/output devices through the window system such that comparison functions can be added, deleted or changed;     wherein the testing system is extensible by modification of the filter functions, save data functions, comparison functions.   
     
     
       2. The user extensible, automated testing system as described in claim 1, wherein the filter modification means further comprises means to create, delete and modify filter functions. 
     
     
       3. The user extensible, automated testing system as described in claim 1, wherein the filter functions, save data functions and comparison functions are selectable by the mouse through menus displayed within windows on the display. 
     
     
       4. The user extensible, automated testing system as described in claim 1, wherein the journaling means generates a journal file by capturing the events representative of user actions and writing the events into the journal file. 
     
     
       5. The user extensible, automated testing system as described in claim 4, wherein the filter functions, save data functions and comparison functions are selectable by the mouse through menus displayed within windows on the display and said journal file comprises the mouse events representative of the user actions to select functions whereby during replay of the journal file the functions will be selected by replay of the events by the journaling means. 
     
     
       6. The user extensible, automated testing system as described in claim 1, wherein said filter means comprises a filter function to eliminate predetermined portions of data. 
     
     
       7. The user extensible, automated testing system as described in claim 1, wherein said filter means comprises a filter function to translate predetermined portions of data from a first format to a second format. 
     
     
       8. The user extensible, automated testing system as described in claim 1, wherein said filter means comprises a plurality of filter functions which are pipelined together to filter data. 
     
     
       9. The user extensible, automated testing system as described in claim 1, wherein said filter modification means comprises a text editor. 
     
     
       10. The user extensible, automated testing system as described in claim 1, wherein said filter execution means is enabled when a save data function is enabled. 
     
     
       11. The user extensible, automated testing system as described in claim 1, wherein said save data functions save text data. 
     
     
       12. The user extensible, automated testing system as described in claim 1, wherein said save data functions save bit map image data. 
     
     
       13. The user extensible, automated testing system as described in claim 1, wherein said save data functions save user-defined data in binary data format. 
     
     
       14. The user extensible, automated testing system as described in claim 1, wherein said save data modification means comprises a text editor. 
     
     
       15. The user extensible, automated testing system as described in claim 1, wherein said comparison means compares two text files. 
     
     
       16. The user extensible, automated testing system as described in claim 1, wherein said comparison means compares two bit map images. 
     
     
       17. The user extensible, automated testing system as described in claim 1, wherein said filter modification means comprises a text editor. 
     
     
       18. The user extensible, automated testing system as described in claim 1, wherein a failure file is generated by storing the save data in the failure file and removing the save data from the failure file if no differences are determined. 
     
     
       19. The user extensible, automated testing system as described in claim 1, wherein said filter execution means is automatically enabled during execution of a save data function. 
     
     
       20. The user extensible, automated testing system as described in claim 1, wherein said comparison means comprises a text editor. 
     
     
       21. The user extensible automated testing system as described in claim 20, the review module further comprises as means to mask specified data coupled to the window system, the master data and the actual data such that the master data and actual data are masked such that the masked data in the actual data is not compared to corresponding master data. 
     
     
       22. The user extensible automated testing system as described in claim 21, wherein the information identifying masks is stored with the corresponding master data. 
     
     
       23. The user extensible automated testing system as described in claim 21 wherein the means to mask specified data highlights the portions of the displayed data which have been masked. 
     
     
       24. The user extensible, automated testing system as described in claim 1, further comprising a review module coupled to the comparison means and the input/output devices through the window system to review the differences between the master data and save data comprising; means to graphically display within a window on the display, the differences between the master data and save data;   whereby the differences between the master data and the save data can be graphically reviewed.   
     
     
       25. The user extensible automated testing system as described in claim 24, wherein the comparison means compares bit map images and the review module further comprises: means to display the actual data as a bit mapped image; and   means to display the differences between the master data and actual data by highlighting the differences in the bit map image display of actual data.   
     
     
       26. The user extensible automated testing system as described in claim 25, wherein the differences are highlighted by turning on and off the pixels of the display which are part of differences. 
     
     
       27. The user extensible automated testing system as described in claim 24, wherein the comparison means compares bit map images and the review module further comprises: means to display the differences between the master data and actual data by displaying only those portions of the image which contain differences.   
     
     
       28. The user extensible automated testing system as described in claim 24, wherein the comparison means compares text data and the review module further comprises: means to display the actual data as lines of text on a first portion of the screen;   means to display the master data as lines of text on a second portion of the screen; and   means to display the differences between the master data and actual data by displaying on a second portion of the display a flag at each line where differences are noted.   
     
     
       29. The user extensible automated testing system as described in claim 24, wherein the comparison means compares text data and the review module further comprises: means to display the actual data as lines of text; and   means to display the differences between the master data and actual data by highlighting the differences in the text display of actual data.   
     
     
       30. The user extensible automated testing system as described in claim 24, wherein the review module further comprises a means to set tolerances on subsequent comparisons such that if the test is replayed and differences are within tolerances set, the comparison will determine a small difference. 
     
     
       31. The user extensible automated testing system as described in claim 30, wherein the tolerances comprise bit shifts of a predetermined number of pixels such that if the pixels are shifted within the predetermined number of pixels, a small difference will be determined. 
     
     
       32. The user extensible automated testing system as described in claim 30, wherein the tolerances comprise color differences of a predetermined numeric value such that if the color of pixels are shifted within the predetemined numeric value, a small difference will be determined. 
     
     
       33. The user extensible automated testing system as described in claim 32, wherein the data is a bit map image and the masked data is identified by designating boundaries of portions of a bit map image displayed on the display. 
     
     
       34. The user extensible automated testing system as described in claim 24, wherein the review module further comprises a means to update the master data being coupled to the actual data and master data, and to the input/output devices through the window system so as to replace the master data with the actual data. 
     
     
       35. The user extensible automated testing system as described in claim 24, wherein said review module further comprises a means to generate and display a log of each comparison performed and whether the comparison passed with no differences determined or failed because of differences determined. 
     
     
       36. The user extensible automated testing system as described in claim 35, wherein said review module further comprises a display selection means which enables the user to select a comparison from the log of comparisons performed, said means controlling the means to graphically display within a window on the display, to generate and display the differences between the master data and save data; whereby the differences between the master data and the save data can be graphically reviewed.   
     
     
       37. The user extensible atuomated testing system as described in claim 24, wherein said review module further comprises a means to set breakpoints in the journal file coupled to said journal file through the window system, said breakpoints causing the journaling means to pause replay of the events recorded in the journal file. 
     
     
       38. The user extensible automated testing system as described in claim 1, further comprising a means to link a predetermined save data function and comparison function coupled to the save data means and the comparison means so as to create a list of save data/comparison function pairs, each save data/comparison pair being identified by name, wherein the save data functions and comparison functions are selected through selection of a named save data/comparison pair. 
     
     
       39. The user extensible automated testing system as described in claim 38, wherein the save data functions and comparison functions are selectable through selection of a save data/comparison pair selectable through a menu displayed within a window on the display. 
     
     
       40. The user extensible automated testing system as described in claim 24, further comprising a means to link a predetermined save data function, comparison function and review function coupled to the save data means, comparison means and review means so as to create a list of save data/comparison/review function triples, each save data/comparison/review triple being identified by name, wherein the save data functions, comparison functions and review functions are selected through selection of a named save data/comparison/review triple. 
     
     
       41. The user extensible automated testing system as described in claim 40, wherein the save data functions, comparison functions and review functions are selectable through selection of a save data/comparison/review triple selectable through a menu displayed within a window on the display. 
     
     
       42. The user extensible automated testing system as described in claim 1 further comprising a means to manually edit the journal file coupled to the window system, said means comprising: means to translate the events in a first format stored in the journal file into a second format translated journal file understandable by the user;   means to add, delete or modify the translated events being coupled to the translated journal file; and   means to translate the translated journal file events from the second format back to the first format, said means being coupled to the translated journal file and the window system;   whereby the user session can be modified to add, delete or change the user actions performed during the user session.   
     
     
       43. The user extensible automated testing system as described in claim 42, further comprising a means to manually create a journal file coupled to the journal file and input/output devices through the window system, said means comprising: means to generate macro routines which defines the parameters of the user interface of the window system the test systems operates within; and   means to include selected macro routines in the journal file to generate events in the journal file representative of user actions.   
     
     
       44. The user extensible automated testing system as described in claim 1, further comprising an interactive editing means coupled to the journal file and the input/output devices through the window system, said means comprising: means to control the journaling means to replay the journal file;   means to halt replay of the journal file;   means to delete the event in the journal file coupled to said halt replay so as to delete the event at which replay was halted; and   means to add events in the journal file coupled to said halt replay so as to add the event at the location at which replay was halted;   whereby a journal file is interactively edited during replay of the journal file.   
     
     
       45. The user extensible automated testing system as described in claim 44, wherein said interactive editing means further comprises: means for marking locations in the journal file; and   means to control the journaling means whereby subsequent replay of the journal file will cause the journaling means to pause replay at the location of the mark;   whereby the means to add events and means to delete events are used to add and delete events at the location of the mark.   
     
     
       46. The user extensible automated testing system as described in claim 44, wherein said interactive editing means reads each event replayed and writes the event to a second journal file, said second journal file being coupled to the means to add events, the means to delete events and the journal file, said means to add events adds events to the second journal file, said means to delete events causes the events to not be written to the second journal file, said second journal file replacing the existing journal file at completion of the interactive edit. 
     
     
       47. In a window-based computer system comprising a central processing unit (CPU), memory, input/output devices comprising a keyboard, mouse and display, wherein the input and output of information is performed through windows defined on the display, a user extensible, automated testing process generating test results for the testing of application programs which compares master data to actual data generated during the testing of the application program, said test results comprising the differences between the master data and actual data, said user extensible, automated process for testing the application program comprising the steps of: generating events representative of user actions performed during a user session, said events comprising events to execute the application program;   generating events to perform comparisons between the master data and actual data comprising the steps of; filtering portions of data which are not determinative of the test results by executing filter functions which filter portions of data, each filter function identified by name and identified in a filter list file, said filter list file comprising a list of names of filter functions available to filter portions of data and is modified to add, delete or change filter functions;   saving data generated during the testing of the application program according to save data functions which save predetermined portions of the data, each save data function identified by a name and enabled by a plurality of events placed in the journal file representative of user actions which selects the save data function, said save data functions are modified to add, delete or change save data functions;   comparing predetermined portions of data from the master data to corresponding portions of data from the actual data, according to comparison functions each identified by name, to determine differences between predetermined portions of the master data and actual data, said comparison functions are modified to add, delete or change comparison functions; and   generating a failure file containing the actual data saved if differences are determined;   generating a journal file by recording events representative of user actions performed during a user session, said user actions comprising actions to execute the application program and actions to perform comparisons between master data and actual data; and   replaying the journal file which causes the events recorded to be executed to test the application program by executing the application program and performing comparisons between the master data and actual data.     
     
     
       48. The process as described in claim 47, further comprising the steps of selecting the filter functions, save data functions and comparison functions by selecting the corresponding menu items on the menus displayed within windows on the display. 
     
     
       49. The process as described in claim 47, wherein the step of generating a journal file comprises capturing the events representative of user actions and writing the events into the journal file. 
     
     
       50. The process as described in claim 47, wherein the step of filtering comprises eliminating predetermined portions of data. 
     
     
       51. The process as described in claim 47, wherein the step of filtering comprises translating predetermined portions of data from a first format to a second format. 
     
     
       52. The process as described in claim 47, wherein the data is text data. 
     
     
       53. The process as described in claim 47, wherein the data is bit map image data. 
     
     
       54. The process as described in claim 47, wherein the data is user-defined data in binary data format. 
     
     
       55. The process as described in claim 47, the step of generating a failure file comprises the steps of storing the save data in the failure file and removing the save data from the failure file if no differences are determined. 
     
     
       56. The process as described in claim 47, further comprising the step of reviewing the differences between the master data and save data by graphically displaying within a window on the display the differences between the master data and save data. 
     
     
       57. The process as described in claim 56, wherein the data are bit map images and the step of reviewing comprises: displaying the actual data as a bit map image; and   displaying the differences between the master data and actual data by highlighting the differences in the bit map image display of actual data.   
     
     
       58. The process as described in claim 57, wherein the differences are highlighted by turning on and off the pixels of the display which are part of differences. 
     
     
       59. The process as described in claim 56, wherein the data are bit map images and the step of reviewing comprises: displaying the differences between the master data and actual data by displaying only those portions of the image which contain differences.   
     
     
       60. The process as described in claim 56, wherein the data is text data and the step of reviewing comprises: displaying the actual data as lines of text on a first portion of the screen;   displaying the master data as lines of text on a second portion of the screen; and   displaying the differences between the master data and actual data by displaying on a second portion of the display a flag at each line where differences are noted.   
     
     
       61. The process as described in claim 56, wherein the data is text data and the step of reviewing comprises: displaying the actual data as lines of text; and   displaying the differences between the master data and actual data by highlighting the differences in the text display of actual data.   
     
     
       62. The process as described in claim 56, wherein the step of reviewing further comprises the step of setting tolerances on subsequent comparisons such that if the test is replayed and differences are are within tolerances set, the comparison will determine a small difference. 
     
     
       63. The process as described in claim 62, wherein the tolerances comprise bit shifts of a predetermined number of pixels such that if the pixels are shifted within the predetermined number of pixels, a small difference will be determined. 
     
     
       64. The process as described in claim 62, wherein the tolerances comprise color differences of a predetermined numeric value such that if the color of pixels are shifted within the predetermined numeric value, a small difference will be determined. 
     
     
       65. The process as described in claim 56, wherein the step of reviewing further comprises the step of masking specified data in the master data and actual data such that the masked data in the actual data is not compared to corresponding master data. 
     
     
       66. The process as described in claim 65, wherein the data is a bit map image and the marked data is identified by designating boundaries of portions of a bit map image displayed on the display. 
     
     
       67. The process as described in claim 65, wherein the information identifying masks is stored with the corresponding master data. 
     
     
       68. The process as described in claim 56, wherein the step of reviewing further comprises the step of updating the master data by replacing the master data with the actual data. 
     
     
       69. The process as described in claim 56, further comprising the step of generating and displaying a log of each comparison performed and whether the comparison passed with no differences determined or failed because of differences determined. 
     
     
       70. The process as described in claim 69, wherein the step of reviewing further comprises the step of selecting a comparison from the log of comparisons performed to generate and display the differences between the master data and save data; whereby the differences between the master data and the save data can be graphically reviewed.   
     
     
       71. The process as described in claim 56, wherein the step of reviewing further comprises the step of setting breakpoints in the journal file, said breakpoints causing the journaling means to pause replay of the events recorded in the journal file 
     
     
       72. The process as described in claim 47, further comprising the step of linking a predetermined save data function and comparison function by creating a list of save data/comparison function pairs, each save data/comparison pair being identified by name, wherein the save data functions and comparison functions are selected through selection of a named save data/comparison pair. 
     
     
       73. The process as described in claim 56, further comprising the step of linking a predetermined save data function comparison function and review function by creating a list of save data/comparison/review function triples, each save data/comparison/review triple being identified by name, wherein the save data functions, comparison functions and review functions are selected through selection of a named save data/comparison/review triple. 
     
     
       74. The process as described in claim 47, further comprising the step of manually editing the journal file, said step comprising: translating the events in a first format stored in the journal file into a second format understandable by the user;   adding, deleting or modifying the translated events; and   translating the translated events from the second format back to the first format;   whereby the user session can be modified to add, delete or change the user actions performed during the user session.   
     
     
       75. The process as described in claim 47, further comprising the step of manually creating a journal file comprising: generating macro routines which define the parameters of the user interface of the window system the test systems operates within; and   including selected macro routines in the journal file to generate events in the journal file representative of user actions.   
     
     
       76. The process as described in claim 47, further comprising the step of interactively editing the journal file, said step comprising: controlling the journaling means to replay the journal file;   halting replay of the journal file;   deleting the event in the journal file at which replay was halted; and   adding events in the journal file at the location at which replay was halted;   whereby a journal file is interactively edited during replay of the journal file.   
     
     
       77. The process as described in claim 76, wherein the step of interactively editing the journal file comprises: marking locations in the journal file; an   controlling the journaling means whereby subsequent replay of the journal file will cause the journaling means to pause replay at the location of the mark;   whereby events are added and deleted at the location of the mark.   
     
     
       78. The process as described in claim 77, wherein the step of interactively editing comprises reading each event replayed and writing the event to a second journal file which subsequently replaces the first journal file, whereby events are added to the second journal file and events are removed from the journal, said means to add events adds events to the second journal file and delete events by not writing to the second journal file.