State machine controlled dynamic distributed computing

Abstract

A computer readable storage medium with executable instructions specifies the execution of a state machine operating across a set of computing nodes in a distributed computing system. The executable instructions execute a set of operators, where the execution of each operator is under the control of a state machine that periodically invokes pause control states to pause the execution of an operator in response to a violation of a service level agreement specifying an operating condition threshold within the distributed computing system. Partitions of input data are formed that are worked on independently within the distributed computing system. A set of data batches associated with the input data is processed. Data partition control states to process the partitions associated with the set of data batches are specified. Key control states to process a set of keys associated with a data partition of the partitions are defined.

1. A computer readable storage medium embodied within a hardware device with executable instructions specifying the execution of a state machine operating across a plurality of computing nodes in a distributed computing system, comprising executable instructions to:
 execute a plurality of operators, wherein the execution of each operator is under the control of a state machine that periodically invokes pause control states to pause the execution of an operator in response to a violation of a service level agreement specifying an operating condition threshold within the distributed computing system; 
 form partitions of input data that are worked on independently within the distributed computing system; 
 process a set of data batches associated with the input data; 
 specify data partition control states to process the partitions associated with the set of data batches; and 
 define key control states to process a set of keys associated with a data partition of the partitions. 
 
     
     
       2. The computer readable storage medium embodied within a hardware device of  claim 1  wherein the state machine stores execution state in response to a pause control state. 
     
     
       3. The computer readable storage medium embodied within a hardware device of  claim 2  wherein the execution state includes partition execution state and key execution state. 
     
     
       4. The computer readable storage medium embodied within a hardware device of  claim 2  wherein the state machine reinitiates execution of a paused operator by invoking the stored execution state. 
     
     
       5. The computer readable storage medium embodied within a hardware device of  claim 4  wherein the state machine reinitiates execution of the paused operator after redistribution of processing tasks on resources in the distributed computing system. 
     
     
       6. The computer readable storage medium embodied within a hardware device of  claim 1  wherein the pause control states include a batch service level agreement check state, a partition service level agreement check state and a key service level agreement check state. 
     
     
       7. The computer readable storage medium embodied within a hardware device of  claim 1  wherein the pause control states include a partition success check state and a key success check state. 
     
     
       8. The computer readable storage medium embodied within a hardware device of  claim 1  wherein the pause control states include a batch service level agreement check state, a partition service level agreement violation check state, a partition pause state, a key service level agreement violation state and a key pause state. 
     
     
       9. The computer readable storage medium embodied within a hardware device of  claim 1  wherein the pause control states include a partition replay status state and a key replay status state. 
     
     
       10. The computer readable storage medium embodied within a hardware device of  claim 1  wherein the pause control states include a partition cleanup state and a key cleanup state. 
     
     
       11. The computer readable storage medium embodied within a hardware device of  claim 1  wherein the state machine is invoked through method calls associated with an application program interface. 
     
     
       12. The computer readable storage medium embodied within a hardware device of  claim 11  wherein the application program interface includes method calls to open a data partition, invoke a key in a data partition and close a data partition. 
     
     
       13. The computer readable storage medium embodied within a hardware device of  claim 11  wherein the application program interface includes method calls to open a key, invoke a subsequent key value and close a key. 
     
     
       14. The computer readable storage medium embodied within a hardware device of  claim 11  wherein the application program interface includes method calls to open batch data, invoke a subsequent batch of data, and close batch data. 
     
     
       15. The computer readable storage medium embodied within a hardware device of  claim 11  wherein the application program interface passes data on service level agreement violations. 
     
     
       16. The computer readable storage medium embodied within a hardware device of  claim 11  wherein the application program interface passes data on input/output exceptions. 
     
     
       17. A method for specifying the execution of a state machine operating across a plurality of computing nodes in a distributed computing system, the method comprising:
 executing a plurality of operators under the control of a state machine configured to periodically invoke pause control states to pause the execution of an operator in response to a violation of a service level agreement specifying an operating condition threshold within the distributed computing system; and 
 forming partitions of input data that are worked on independently within the distributed computing system. 
 
     
     
       18. The method of  claim 17 , further comprising:
 processing a set of data batches associated with the input data; and 
 specifying data partition control states to process the partitions associated with the set of data batches. 
 
     
     
       19. The method of  claim 18 , further comprising:
 defining key control states to process a set of keys associated with a data partition of the specified data partitions. 
 
     
     
       20. The method of  claim 18  wherein the state machine is further configured to check for a violation of a service level agreement. 
     
     
       21. The method of  claim 18  wherein the pause control states comprise a partition pause state. 
     
     
       22. The method of  claim 18  wherein the pause control states comprise a key pause state. 
     
     
       23. The method of  claim 17  wherein the state machine is further configured to store execution state in response to a pause control state. 
     
     
       24. The method of  claim 23  wherein the state machine is further configured to reinitiate execution of a pause operator by invoking the stored execution state. 
     
     
       25. A computer system, including a memory and a processor, the computer system comprising:
 a component configured to execute a plurality of operators under the control of a state machine, the state machine being configured to periodically invoke pause control states to pause the execution of an operator in response to a violation of a service level agreement specifying an operating condition threshold within the distributed computing system; and 
 a component configured to form partitions of input data that are worked on independently within the distributed computing system, 
 wherein at least one of the components comprises computer-executable instructions stored in the memory for execution by the computer system. 
 
     
     
       26. The computer system of  claim 25 , further comprising:
 a component configured to process a set of data batches associated with the input data; 
 a component configured to specify data partition control states to process the partitions associated with the set of data batches; and 
 a component configured to define key control states to process a set of keys associated with a data partition of the specified data partitions. 
 
     
     
       27. The computer system of  claim 25  wherein the pause control states include a batch service level agreement check state, a partition service level agreement violation check state, a partition pause state, a key service level agreement violation state or a key pause state. 
     
     
       28. The computer system of  claim 25  wherein the state machine is further configured to check for a violation of a service level agreement. 
     
     
       29. The computer system of  claim 25  wherein the pause control states comprise a partition pause state and a key pause state.