Managing extract-transform-load operations

Abstract

Managing ETL (extract-transform-load) operation execution by receiving an updated ETL (extract-transform-load) job flow, determining affected stages of a current ETL job flow, sending a message to stop data flow for all stages of the current ETL job flow, sending a switch message along the current ETL job flow through the affected stages, receiving notice of receipt of the switch message downstream from the affected stages of the current ETL job flow, moving an affected stage of the current ETL job flow from a first ETL section to a second ETL section; and resuming data flow according to the updated ETL job flow.

What is claimed is: 
     
       1. A computer implemented method for managing ETL (extract-transform-load) operation execution, the method comprising:
 receiving an updated ETL (extract-transform-load) job flow; 
 identifying a first section of the current ETL flow for replacement; 
 instantiating a second section according to the updated ETL flow; 
 ceasing data flow through the first section; 
 sending a switch message through the first section of the first ETL flow; 
 receiving notice that the switch message has passed through the first section of the first ETL flow; 
 replacing the first section of the first ETL flow with the second section; 
 determining stages of a current ETL job flow affected by the updated ETL job flow; 
 sending a message to stop data flow for all stages of the current ETL job flow unaffected by the updated ETL job flow; 
 sending a switch message along the current ETL job flow through the stages affected by the updated ETL job flow; 
 receiving notice of receipt of the switch message downstream from the stages of the current ETL job flow affected by the updated ETL job flow; 
 moving a stage of the current ETL job flow, affected by the updated ETL job flow, from one side of a remote stage to another side of the remote stage; and 
 resuming data flow according to the updated ETL job flow. 
 
     
     
       2. The method of  claim 1 , wherein moving an affected stage includes moving to a new virtual machine. 
     
     
       3. The method of  claim 1 , wherein moving an affected stage includes moving to a new cloud environment. 
     
     
       4. The method of  claim 1 , wherein the current and updated ETL flows are embodied in non-cyclic graphs. 
     
     
       5. The method according to  claim 1 , further comprising:
 stopping data flow from all other first ETL flow stages; 
 sending, by a first ETL flow stage, a switch message along the first ETL flow; 
 receiving notice of receipt of the switch message at a first ETL flow destination stage; 
 constructing a new ETL flow; and 
 starting data flow in the new ETL flow. 
 
     
     
       6. The method according to  claim 1 , wherein moving the affected stage of the current ETL job flow from a first ETL section to a second ETL section, comprises moving the affected stage upstream. 
     
     
       7. A computer program product for managing ETL (extract-transform-load) operation execution, the computer program product comprising one or more computer readable storage devices and program instructions collectively stored on the one or more computer readable storage devices, the stored program instructions comprising:
 program instructions to receive an updated ETL (extract-transform-load) job flow; 
 program instructions to identify a first section of the current ETL flow for replacement; 
 program instructions to instantiate a second section according to the updated ETL flow; 
 program instructions to cease data flow through the first section; 
 program instructions to send a switch message through the first section of the first ETL flow; 
 program instructions to receive notice that the switch message has passed through the first section of the first ETL flow; 
 program instructions to replace the first section of the first ETL flow with the second section 
 program instructions to determine stages of a current ETL job flow affected by the updated ETL job flow; 
 program instructions to send a message to stop data flow for all stages of the current ETL job flow unaffected by the updated ETL job flow; 
 program instructions to send a switch message along the current ETL job flow through the stages affected by the updated ETL job flow; 
 program instructions to receive notice of receipt of the switch message downstream from the stages of the current ETL job flow affected by the updated ETL job flow; 
 program instructions to move a stage of the current ETL job flow affected by the updated ETL job flow, from one side of a remote stage to another side of the remote stage; and 
 program instructions to resume data flow according to the updated ETL job flow. 
 
     
     
       8. The computer program product according to  claim 7 , wherein moving an affected stage comprises moving to a new virtual machine. 
     
     
       9. The computer program product according to  claim 7 , wherein moving the affected stage comprises moving to a new cloud environment. 
     
     
       10. The computer program product according to  claim 7 , wherein the current and updated ETL flows are embodied in non-cyclic graphs. 
     
     
       11. The computer program product according to  claim 7 , the stored program instructions further comprising:
 program instructions to stop data flow from all other first ETL flow stages; 
 program instructions to send, by a first ETL flow stage, a switch message along the first ETL flow; 
 program instructions to receive notice of receipt of the switch message at a first ETL flow destination stage; 
 program instructions to construct a new ETL flow; and 
 program instructions to start data flow in the new ETL flow. 
 
     
     
       12. The computer program product according to  claim 7 , wherein moving the affected stage of the current ETL job flow from a first ETL section to a second ETL section, comprises moving the affected stage upstream. 
     
     
       13. A computer system for managing ETL (extract-transform-load) operation execution, the computer system comprising:
 one or more computer processors; 
 one or more computer readable storage devices; and 
 stored program instructions on the one or more computer readable storage devices for execution by the one or more computer processors, the stored program instructions comprising: 
 program instructions to receive an updated ETL (extract-transform-load) job flow; 
 program instructions to identify a first section of the current ETL flow for replacement; 
 program instructions to instantiate a second section according to the updated ETL flow; 
 program instructions to cease data flow through the first section; 
 program instructions to send a switch message through the first section of the first ETL flow; 
 program instructions to receive notice that the switch message has passed through the first section of the first ETL flow; and 
 program instructions to replace the first section of the first ETL flow with the second section 
 program instructions to determine stages of a current ETL job flow affected by the updated ETL job flow; 
 program instructions to send a message to stop data flow for all stages of the current ETL job flow unaffected by the updated ETL job flow; 
 program instructions to send a switch message along the current ETL job flow through the stages affected by the updated ETL job flow; 
 program instructions to receive notice of receipt of the switch message downstream from the stages of the current ETL job flow affected by the updated ETL job flow; 
 program instructions to move a stage of the current ETL job flow affected by the updated ETL job flow, from one side of a remote stage to another side of the remote stage; and 
 program instructions to resume data flow according to the updated ETL job flow. 
 
     
     
       14. The computer system according to  claim 13 , wherein moving an affected stage comprises moving to a new virtual machine. 
     
     
       15. The computer system according to  claim 13 , wherein moving the affected stage comprises moving to a new cloud environment. 
     
     
       16. The computer system according to  claim 13 , wherein the current and updated ETL flows are embodied in non-cyclic graphs. 
     
     
       17. The computer system according to  claim 13 , the stored program instructions further comprising:
 program instructions to stop data flow from all other first ETL flow stages; 
 program instructions to send, by a first ETL flow stage, a switch message along the first ETL flow; 
 program instructions to receive notice of receipt of the switch message at a first ETL flow destination stage; 
 program instructions to construct a new ETL flow; and 
 program instructions to start data flow in the new ETL flow.