Data processing method and apparatus

Abstract

In a data processing method, a worker node in a distributed data processing system receives first data from an upstream worker node. The first data has been stored in a buffer of the upstream worker node. The worker node sends a first portion of the first data to a persistent storage device of the distributed data processing system for persistent backup, and performs computational processing on the first data to generate second data. Prior to completing performing computational processing on the first data, the worker node sends acknowledgement information to the upstream worker node to instruct the upstream node to delete the first data from the buffer of the upstream worker node. The worker node then sends the second data to a downstream worker node in the distributed data processing system for further processing by the downstream worker node.

What is claimed is: 
     
       1. A data processing method performed by a worker node in a distributed data processing system including a plurality of worker nodes and a persistent storage device, the worker node including a compute thread, comprising:
 receiving first data from an upstream worker node, the first data being stored in a local buffer of the upstream worker node; 
 allocating the received first data to the compute thread for computational processing; 
 sending a first portion of the first data to the persistent storage device of the distributed data processing system for persistent backup, wherein the persistent storage device is configured to communicate with all the plurality of worker nodes and to receive and store data or state information generated from each of the plurality of worker nodes when the respective worker node performs computational processing; 
 performing the computational processing on the first data by the compute thread to generate second data; 
 after allocating the received first data to the compute thread and prior to completing performing the computational processing on the first data, sending acknowledgement information to the upstream worker node to instruct the upstream worker node to delete the first data from the local buffer of the upstream worker node; and 
 sending the second data to a downstream worker node in the distributed data processing system for processing by the downstream worker node, 
 wherein the plurality of worker nodes includes the upstream worker node, the worker node and the downstream worker node. 
 
     
     
       2. The method according to  claim 1 , further comprising, after receiving the first data, buffering the first data in a local buffer of the worker node up to a first quantity,
 wherein the first portion of the first data sent to the persistent storage device comprises a remainder of the first data that is not buffered in the local buffer. 
 
     
     
       3. The method according to  claim 1 , wherein performing the computational processing on the first data by the compute thread comprises:
 generating multiple pieces of intermediate state information corresponding to multiple pieces of intermediate state information in processing the first data, wherein each piece of the multiple pieces of intermediate state information comprises an intermediate result obtained by the worker node; and 
 sending the multiple pieces of intermediate state information to the persistence storage device for persistent backup. 
 
     
     
       4. The method according to  claim 3 , wherein sending the multiple pieces of intermediate state information to the persistent storage device comprises:
 determining that a divergence between a current intermediate result corresponding to current intermediate state information and a latest stored intermediate result already stored in the persistence storage device is greater than a divergence threshold; and 
 sending the current intermediate state information to the persistence storage device for persistent backup. 
 
     
     
       5. The method according to  claim 2 , wherein sending the first portion of the first data to the persistent storage device for persistent backup comprises:
 determining whether an amount of the first data buffered in the local buffer of the worker node is equal to or larger than the first quantity; and 
 upon determination that the amount of the first data buffered in the local buffer of the worker node is equal to or larger than the first quantity, sending the first portion of the first data to the persistent storage device for persistent backup, the first portion of the first data being the remainder of the first data that is not buffered in the local buffer. 
 
     
     
       6. A worker node in a distributed data processing system including a plurality of worker nodes and a persistent storage device, the worker node including a compute thread, comprising;
 a memory storing computer-executable instructions; and 
 at least one processor configured to execute the computer-executable instructions to perform operations comprising:
 receiving first data from an upstream worker node in the distributed data processing system, the first data being stored in a local buffer of the upstream worker node; 
 allocating the received first data to the compute thread for computational processing, 
 sending a first portion of the first data to the persistent storage device of the distributed data processing system for persistent backup, wherein the persistent storage device is configured to communicate with all the plurality of worker nodes and to receive and store data or state information generated from each of the plurality of worker nodes when the respective worker node performs computational processing; 
 performing the computational processing on the first data by the compute thread to generate second data; 
 after allocating the received first data to the compute thread and prior to completing performing the computational processing on the first data, sending acknowledgement information to the upstream worker node to instruct the upstream worker node to delete the first data from the local buffer of the upstream worker node; and 
 
 sending the second data to a downstream worker node in the distributed data processing system for processing by the downstream worker node, 
 wherein the plurality of worker nodes includes the upstream worker node, the worker node and the downstream worker node. 
 
     
     
       7. The worker node according to  claim 6 , wherein the operations further comprise buffering the first data in a local buffer of the worker node up to a first quantity, and the first portion of the first data sent to the persistent storage device comprises a remainder of the first data that is not buffered in the local buffer. 
     
     
       8. The worker node according to  claim 6 , wherein the operations further comprise:
 generating multiple pieces of intermediate state information corresponding to multiple pieces of intermediate state information in processing the first data, wherein each piece of the multiple pieces of intermediate state information comprises an intermediate result obtained by the worker node; and 
 sending the multiple pieces of intermediate state information to the persistence storage device for persistent backup. 
 
     
     
       9. The worker node according to  claim 8 , wherein the operations further comprise:
 determining that a divergence between a current intermediate result corresponding to current intermediate state information and a latest stored intermediate state result already stored in the persistence storage device is greater than a divergence threshold; and 
 sending the current intermediate state information to the persistence storage device for persistent backup. 
 
     
     
       10. The worker node according to  claim 6 , wherein the first data is received from the upstream worker node via an intra-process communication protocol or an inter-process communication protocol. 
     
     
       11. The worker node according to  claim 7 , wherein sending the first portion of the first data to the persistent storage device for persistent backup comprises:
 determining whether an amount of the first data buffered in the local buffer of the worker node is equal to or larger than the first quantity; and 
 upon determination that the amount of the first data buffered in the local buffer of the worker node is equal to or larger than the first quantity, sending the first portion of the first data to the persistent storage device for persistent backup, the first portion of the first data being the remainder of the first data that is not buffered in the local buffer. 
 
     
     
       12. The worker node according to  claim 7 , wherein the operations further comprise:
 determining whether a quantity of failures occurred on the worker node within a time period is larger than a failure threshold, and 
 upon determination that the quantity of failures occurred on the worker node within the time period is larger than the failure threshold, decreasing the first quantity. 
 
     
     
       13. The worker node according to  claim 9 , wherein the operations further comprise:
 determining whether a quantity of failures occurred on the worker node within a time period is larger than a failure threshold, and 
 upon determination that the quantity of failures occurred on the worker node within the time period is larger than the failure threshold, decreasing the divergence threshold.