Testing framework with dynamic applicability management

Abstract

Embodiments of the invention are directed to systems, methods, and devices for providing dynamic applicability management with respect to a testing framework. Executable code segments can be provided in/with various test procedures that include instructions for testing hardware, software, capabilities, features, functionalities, or protocols of a computing device. Each code segment can encapsulate logic that, when executed, identifies whether a corresponding test procedure is applicable to a device given that device's configuration. By executing each code segment, a set of applicable test procedures can be identified. Identifiers or the test procedures may be provided to the device to be tested or a testing platform configured to conduct the test of that device. Transmitting the identifiers and/or procedures configures those devices to perform the test through simulating a legitimate data exchange.

What is claimed is: 
     
         1 . A computer-implemented method comprising:
 obtaining, by a first computing device comprising a processor and a memory, a plurality of executable code segments of a plurality of test procedures that are associated with a plurality of source code providers, each of the plurality of executable code segments comprising a respective function having a common function name and a common input parameter list, each of the plurality of executable code segments being configured to receive, when executed, a respective set of device configuration parameters and output an indication that a corresponding test procedure of the plurality of test procedures is applicable to the set of input parameters;   receiving, by the first computing device, a set of device configuration parameters associated with a second computing device;   executing, by the first computing device using a plurality of functions sharing the common function name, the plurality of executable code segments, the plurality of executable code segments being individually provided the set of device configuration parameters associated with the second computing device as input;   determining, by the first computing device from outputs returned from executing the plurality of executable code segments, a subset of outputs that indicate that a subset of the plurality of test procedures are applicable to the set of device configuration parameters; and   causing a receiving device comprising a corresponding processor and a corresponding memory to execute the subset of the plurality of test procedures based at least in part on transmitting, by the first computing device to the receiving device, applicability data that identifies the subset of the plurality of test procedures, the receiving device being configured to request or initiate execution of the subset of the plurality of test procedures in response to receiving the applicability data from the first computing device.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the set of device configuration parameters indicate one or more features supported by the second computing device. 
     
     
         3 . The computer-implemented method of  claim 1 , wherein each of the plurality of executable code segments are obtained from a respective test procedure of the plurality of test procedures. 
     
     
         4 . The computer-implemented method of  claim 3 , wherein each of the plurality of test procedures comprises corresponding machine-readable data identifying a set of instructions for performing a test or a simulation. 
     
     
         5 . The computer-implemented method of  claim 4 , further comprising transmitting, to one or more computing devices, one or more machine-readable data instances comprising corresponding sets of instructions that individually correspond to the subset of the plurality of test procedures. 
     
     
         6 . The computer-implemented method of  claim 5 , wherein the one or more computing devices comprise the second computing device and a testing platform computer, and wherein transmitting the one or more machine-readable data instances configures the second computing device and the testing platform computer to perform the simulation, the simulation testing functionality provided by the second computing device, the functionality corresponding to one or more capabilities purported to be supported by the second computing device. 
     
     
         7 . The computer-implemented method of  claim 6 , wherein each of the plurality of test procedures comprises configuration data for configuring at least one of the second computing device or the testing platform computer to perform the simulation. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein the applicability data comprises the subset of the plurality of test procedures. 
     
     
         9 . The computer-implemented method of  claim 1 , wherein the plurality of test procedures are provided by one or more test procedure provider computers. 
     
     
         10 . The computer-implemented method of  claim 9 , wherein each of the one or more test procedure provider computers are associated with differing entities. 
     
     
         11 . A management computer, comprising:
 one or more processors; and   one or more memories that store computer-executable instructions that, when executed by the one or more processors, cause the management computer to:
 obtain a plurality of executable code segments of a plurality of test procedures that are associated with a plurality of source code providers, each of the plurality of executable code segments comprising a respective function having a common function name and a common input parameter list, each of the plurality of executable code segments being configured to receive, when executed, a respective set of device configuration parameters and output an indication that a corresponding test procedure of the plurality of test procedures is applicable to the set of input parameters; 
 receive a set of device configuration parameters associated with a computing device; 
 execute, using a plurality of functions sharing the common function name, the plurality of executable code segments, the plurality of executable code segments being individually provided the set of device configuration parameters associated with the computing device as input; 
 determine, from outputs obtained from executing the plurality of executable code segments, a subset of outputs that indicate that a subset of the plurality of test procedures are applicable to the set of device configuration parameters; and 
 cause a receiving device comprising a corresponding processor and a corresponding memory to execute the subset of the plurality of test procedures based at least in part on transmitting to the receiving device, applicability data that identifies the subset of the plurality of test procedures, the receiving device being configured to request or initiate execution of the subset of the plurality of test procedures in response to receiving the applicability data. 
   
     
     
         12 . The management computer of  claim 11 , wherein the set of device configuration parameters comprise at least one of: a hardware component, a software component, a feature, a capability, or a protocol associated with a configuration of the computing device. 
     
     
         13 . The management computer of  claim 11 , wherein executing the computer-executable instructions further causes the management computer to:
 receive the plurality of test procedures from the plurality of source code providers, the plurality of executable code segments being provided within the plurality of test procedures; and   store the plurality of test procedures for subsequent use.   
     
     
         14 . The management computer of  claim 11 , wherein the set of device configuration parameters are received by at least one of: the computing device to be tested or a testing platform computer that is configured to execute one or more tests with the computing device to be tested. 
     
     
         15 . The management computer of  claim 11 , wherein executing the plurality of executable code segments associated with the plurality of source code providers comprises:
 accessing a first test procedure of the plurality of test procedures;   executing a first code segment of the first test procedure;   storing a first output obtained from executing the first code segment;   accessing a second test procedure of the plurality of test procedures;   executing a second code segment of the second test procedure; and   storing a second output obtained from executing the second code segment.   
     
     
         16 . The management computer of  claim 11 , wherein the receiving device that receives the applicability data is a testing platform computer that initiates a test with the computing device based at least in part on receiving the applicability data. 
     
     
         17 . The management computer of  claim 11 , wherein a given test procedure of the plurality of test procedures specifies one or more configuration settings for a respective computing device that participates in a corresponding test. 
     
     
         18 . The management computer of  claim 11 , wherein each of the plurality of executable code segments encapsulates conditional logic for determining a corresponding test procedures applicability with respect to a device configuration of the computing device, the device configuration being specified by the set of device configuration parameters. 
     
     
         19 . The management computer of  claim 11 , wherein at least one test procedure of the plurality of test procedures is associated with executing a simulation using a testing platform computer to simulate a legitimate data exchange with the computing device.