Testing and mitigation framework for networked devices

Abstract

The present disclosure generally relates to the automated testing of a system that includes software or hardware components. In some embodiments, a testing framework generates a set of test cases for a system under test using a grammar. Each test case may perform an action, such as provide an input to the system under test, and result in an output from the system under test. The inputs and outputs are then compared to the expected results to determine whether the system under test is performing correctly. The data can then be interpreted in the grammar system or used as input to a fault isolation engine to determine anomalies in the system under test. Based on identified faults, one or more mitigation techniques may be implemented in an automated fashion.

What is claimed is: 
     
         1 . A testing apparatus comprising:
 a data processing unit comprising one or more processors, the data processing unit operative to assess a functionality of a device under test based, at least partly, on control instructions generated in accordance with a testing grammar; and   a mitigation unit comprising one or more processors, the mitigation unit operative to:
 in response to a fault of the device under test as assessed by the data processing unit, identify a grammar production from which at least a portion of the control instructions are derived, wherein the testing grammar includes the grammar production; and 
 cause performance of at least one mitigation technique corresponding to the functionality of the device under test based, at least in part, on the grammar production, the mitigation technique utilized to limit the functionality of the device under test. 
   
     
     
         2 . The testing apparatus of  claim 1 , wherein the testing grammar further includes a set of terminal symbols and a set of non-terminal symbols. 
     
     
         3 . The testing apparatus of  claim 2 , wherein the grammar production includes rules for grouping the terminal or non-terminal symbols. 
     
     
         4 . The testing apparatus of  claim 1 , wherein the device under test corresponds to a programmable logic controller (PLC). 
     
     
         5 . The testing apparatus of  claim 1 , wherein the data processing unit assesses the fault of the device under test based, at least in part, on an output from the device under test. 
     
     
         6 . The testing apparatus of  claim 5 , wherein the output from the device under test is generated in response to at least a portion of the control instructions. 
     
     
         7 . The testing apparatus of  claim 5 , wherein the data processing unit assesses the fault of the device under test based further on a comparison between the output from the device under test and an expected output. 
     
     
         8 . The testing apparatus of  claim 1 , wherein the control instructions correspond to at least one of internet control message protocol messages or a set of test cases generated based at least partly on input from a client. 
     
     
         9 . The testing apparatus of  claim 1 , wherein the mitigation technique corresponds to a configuration to prevent transmission of one or more control instructions derived from the grammar production. 
     
     
         10 . The testing apparatus of  claim 9 , wherein the configuration corresponds to a configuration of a firewall. 
     
     
         11 . A computer implemented method comprising:
 obtaining identification of assessed performance of a system under test;   identifying a target grammar production as a grammar production that elicited a fault or failure in accordance with the assessed performance of the system under test; and   causing performance of at least one mitigation technique with respect to the system under test based, at least in part, on the target grammar production,   the method performed programmatically by one or more computing systems under control of executable program code.   
     
     
         12 . The method of  claim 11 , wherein the identification of assessed performance of the system under test comprises at least one notation of a fault condition. 
     
     
         13 . The method of  claim 11 , wherein identifying the target grammar production comprises isolating one or more target control instructions transmitted to the system under test. 
     
     
         14 . The method of  claim 13 , wherein isolating one or more target control instructions is based, at least in part, on a probabilistic criterion. 
     
     
         15 . The method of  claim 14 , wherein the probabilistic criterion is associated with a Hidden Markov Model. 
     
     
         16 . The method of  claim 13 , wherein isolating one or more target control instructions comprises transmitting the one or more target control instructions to the system under test for verification of an effect of the one or more target control instructions. 
     
     
         17 . A non-transitory computer readable storage medium storing computer executable instructions that direct a computer system comprising one or more processors to perform operations comprising:
 obtaining indication of assessed performance of a system under test;   identifying a first target grammar production as a grammar production that elicited a fault or failure in accordance with the assessed performance of the system under test; and   causing performance of at least one mitigation technique with respect to the system under test based, at least in part, on the target grammar production.   
     
     
         18 . The non-transitory computer readable storage medium of  claim 17 , wherein the operations further comprise identifying a second target grammar production as a grammar production that elicited the fault or failure in combination with the first target grammar production. 
     
     
         19 . The non-transitory computer readable storage medium of  claim 17 , wherein identifying the first target grammar production comprises encoding of grammar productions into numbers. 
     
     
         20 . The non-transitory computer readable storage medium of  claim 17 , wherein identifying the first target grammar production comprises identifying the first target grammar production in accordance with a parse tree.