Multi-layer test suite generation

Abstract

A system and method for developing a multi-layered test suite for an operating platform including a framework layer and a system layer includes executing a first test suite against a version of the operating platform modified based on a software faults. A first counter is incremented if a first test suite executed against the modified version of the operating platform fails. A second test suite can be executed against the modified version of the operating platform and test cases may be added to the first test suite based on whether the second test suite passes or fails.

1 . A method for developing a test suite for an operating platform, the method comprising:
 performing a plurality of iterations, each of the plurality of iterations corresponding to a selected one of a plurality of software faults, wherein each iteration includes:
 executing a first test suite against a modified version of the operating platform corresponding to the selected one of the plurality of software faults, the first test suite including a first plurality of test cases, 
 incrementing a first counter if executing the first test suite against the modified version of the operating platform results in at least one of the first plurality of test cases failing, 
 executing a second test suite against the modified version of the operating platform, wherein the second test suite includes a second plurality of test cases, 
 incrementing a second counter if executing the second test suite against the modified version of the operating platform results in at least one of the second plurality of test cases failing, 
 adding a first test case to the first plurality of test cases when executing the second test suite against the modified version of the operating platform results in at least one of the second plurality of test cases failing, the first test case corresponding to the selected software fault of the iteration, 
 incrementing a third counter if executing the second test suite against the modified version of the operating platform results in none of the second plurality of test cases failing, 
 determining a ratio of the first counter to a sum of the second and third counters, 
 performing another iteration of the plurality of iterations if the ratio fails to satisfy a first threshold, and 
   ceasing iteration through the plurality of iterations if the ratio satisfies the first threshold.   
     
     
         2 . The method of  claim 1  wherein executing the second test suite against the modified version of the operating platform occurs if none of the first plurality of test cases fails when executing the first test suite against the modified version of the operating platform. 
     
     
         3 . (canceled) 
     
     
         4 . The method of  claim 1  wherein adding the first test case to the first plurality of test cases occurs prior to performing the next iteration of the plurality of iterations. 
     
     
         5 . The method of  claim 1  further comprising flagging, in another iteration, the selected one of the plurality of software faults if executing the second test suite against the modified version of the operating platform results in at least one of the second plurality of test cases failing. 
     
     
         6 . The method of  claim 5  further comprising developing a second test case based on the flagged selected one of the plurality of software faults. 
     
     
         7 . The method of  claim 6  further comprising adding the second test case to the first plurality of test cases based on the flagged selected one of the plurality of software faults. 
     
     
         8 . The method of  claim 1  wherein executing the first test suite against the modified version of the operating platform is performed if the first test suite includes at least one test case. 
     
     
         9 . The method of  claim 1  wherein performing the next iteration is further based on whether a quantity of previously performed iterations fails to satisfy a second threshold. 
     
     
         10 . The method of  claim 1  wherein the modified version of the operating platform has been created by altering a baseline version of the operating platform to include the selected one of a plurality of software faults. 
     
     
         11 . A system for developing a test suite for an operating platform, the system comprising:
 a processor; and,   a memory storing instructions that when executed by the processor, cause the processor to perform a plurality of iterative operations, wherein each of the plurality of iterative operations includes:
 executing a first test suite against a modified version of the operating platform, wherein the modified version of the operating platform is created by altering a baseline version of the operating platform; 
 incrementing a first counter if executing the first test suite against the modified version of the operating platform fails, 
 executing a second test suite against the modified version of the operating platform if the first test suite against the modified version of the operating platform passes, 
 incrementing a second counter when executing the second test suite against the modified version of the operating platform fails, 
 adding a test case to the first test suite when executing the second test suite against the modified version of the operating platform results in the second test suite failing, the test case corresponding to the modified version of the operating platform, 
 incrementing a third counter if executing the second test suite against the modified version of the operating platform passes, 
 determining a ratio of the first counter to a sum of the second and third counters, 
 performing another of the plurality of iterative operations if the ratio fails to satisfy a first threshold, and 
 ceasing performance of the plurality of iterative operations if the ratio satisfies the first threshold. 
   
     
     
         12 . (canceled) 
     
     
         13 . The system of  claim 11  wherein adding the test case to the first test suite occurs prior to performing another of the plurality of iterative operations. 
     
     
         14 . The system of  claim 11  wherein executing the first test suite against the modified version of the operating platform is performed if the first test suite includes at least one test case. 
     
     
         15 . The system of  claim 11  wherein performing another of the plurality of iterative operations is further based on whether a quantity of previously performed iterative operations of the plurality of iterative operations fails to satisfy a second threshold. 
     
     
         16 . The system of  claim 11  wherein altering the baseline version of the operating platform is performed by injecting one of a plurality of software faults into the baseline version. 
     
     
         17 . The system of  claim 16  wherein each of the plurality of iterative operations corresponds to each of the plurality of software faults. 
     
     
         18 . A method for creating a test suite for an operating platform, the method comprising:
 performing a plurality of iterations corresponding to a plurality of software faults, wherein each of the plurality of iterations corresponds to one of the plurality of software faults, and each of the plurality of iterations includes:
 creating a modified version of the operating platform by altering a baseline version of the operating platform by injecting a corresponding one of the plurality of software faults into the baseline version of the operating platform; 
 executing a first test suite against the modified version of the operating platform if the first test suite includes at least one test case and incrementing a first counter if executing the first test suite against the modified version of the operating platform fails; and, 
 if executing the first test suite against the modified version of the operating platform passes:
 executing a second test suite against the modified version of the operating platform, 
 incrementing a second counter if executing the second test suite against the modified version of the operating platform fails, and 
 adding a new test case to the first test suite for the corresponding one of the plurality of software faults if executing the second test suite against the modified version of the operating platform fails; 
 
 incrementing a third counter if executing the second test suite against the modified version of the operating platform passes; 
 determining a ratio of the first counter to a sum of the second and third counters; 
 performing another of the plurality of iterations based on if the ratio fails to satisfy a first threshold, and 
 ceasing performance of the plurality of iterations if the ratio satisfies the first threshold. 
   
     
     
         19 . The method of  claim 18  wherein the performing another of the plurality of iterations is further based on whether a quantity of previously performed iterations of the plurality of iterations fails to satisfy a second threshold. 
     
     
         20 . The method of  claim 19  wherein the second threshold is based on a ratio of the previously performed iterations of the plurality of iterations to a total of the plurality of iterations. 
     
     
         21 . The method of  claim 1  wherein the operating platform has a framework layer and a system layer, wherein the selected software fault is injected into the system layer, wherein the second plurality of test cases is configured for testing the framework layer and the system layer, the method further comprising storing the first test suite, after ceasing the iteration through the plurality of iterations, as a test suite for the operating platform, the stored test suite configured for testing the system layer separately from the framework layer. 
     
     
         22 . The system of  claim 11  wherein the operating platform has a framework layer and a system layer, wherein a selected software fault is injected into the system layer, wherein the second test suite is configured for testing the framework layer and the system layer, the plurality of iterative operations further comprising storing the first test suite, after ceasing the performance of the plurality of iterative operations, as a test suite for the operating platform, the stored test suite configured for testing the system layer separately from the framework layer.