Trace messaging for distributed execution of data processing pipelines

Abstract

A method for routing trace messages during an execution of a data processing pipeline is provided. The method may include receiving, at a master node, a request from a client to receive trace messages output by a worker node executing at least a portion of the data processing pipeline. The master node and the worker node may be part of a distributed cluster of computing nodes. The trace messages may correspond to events occurring during the execution of at least the portion of the data processing pipeline. The master node may respond to the request by subscribing to a trace stream published by the worker node. The trace stream may include the trace messages output by the worker node. The master node may generate a user interface for displaying, at the client, the trace messages. Related systems and articles of manufacture, including computer program products, are also provided.

What is claimed is: 
     
       1. A system, comprising:
 at least one data processor; and 
 at least one memory storing instructions which, when executed by the at least one data processor, result in operations comprising:
 receiving, at a first master node, a request from a client to receive a first stream including one or more trace messages output by a first worker node executing at least a portion of a data processing pipeline, the data processing pipeline including a sequence of data processing operations, the first master node and the first worker node comprising a distributed cluster of computing nodes that further includes a second worker node, and the one or more trace messages included in the first trace stream corresponding to one or more events occurring during the execution of at least the portion of the data processing pipeline; 
 in response to the request from the client, subscribing, by the first master node, to the first trace stream published by the first worker node, the first master node subscribing, based at least on the client requesting to receive the first trace stream published by the first worker node but not a second trace stream published by the second worker node, to the first trace stream published by the first worker node but not the second trace stream published by the second worker node; 
 in response to a failure at the first master node, subscribing, by a second master node, to the first trace stream published by the first worker node; and 
 generating, by the first master node or the second master node, a user interface for displaying, at the client, the one or more trace messages included in the first trace stream published by the first worker node. 
 
 
     
     
       2. The system of  claim 1 , wherein the first worker node executes a first portion of the data processing pipeline and the second worker node executes a second portion of the data processing pipeline, and wherein the first master node coordinates the execution of the data processing pipeline by the first worker node and the second worker node. 
     
     
       3. The system of  claim 2 , further comprising:
 receiving, from the client, another request to receive a second trace stream including one or more trace messages output by the second worker node during the execution of the second portion of the data processing pipeline; and 
 responding to the other request by at least subscribing to the second trace stream published by the second worker node. 
 
     
     
       4. The system of  claim 3 , further comprising:
 storing, in a trace log, at least one the one or more trace messages included in the first trace stream published by the first worker node and the one or more trace messages included in the second trace stream published by the second worker node. 
 
     
     
       5. The system of  claim 2 , wherein the first portion of the data processing pipeline and the second portion of the data processing pipeline each include one or more data processing operations from the sequence of data processing operations, and wherein the one or more data processing operations are performed on data stored in a database. 
     
     
       6. The system of  claim 1 , wherein a listener is deployed at the first worker node in response to the first master node subscribing to the first trace stream published by the first worker node, wherein the listener is configured to detect when the first worker node outputs a trace message, and wherein at least some trace messages output by the first worker node are pushed to the first master node. 
     
     
       7. The system of  claim 6 , wherein the one or more trace messages output by the first worker node are filtered prior to being pushed to the first master node, and wherein the one or more trace messages are filtered based at least on a threshold severity level specified by the client. 
     
     
       8. The system of  claim 1 , wherein the data processing pipeline is associated with a graph representative of the data processing pipeline, wherein the graph includes a plurality of nodes corresponding to the sequence of data processing operations, wherein the graph further includes one or more edges indicating a flow of data between consecutive data processing operations, and wherein at least the portion of the data processing pipeline is executed based at least on the graph. 
     
     
       9. A computer-implemented method, comprising:
 receiving, at a first master node, a request from a client to receive a first stream including one or more trace messages output by a first worker node executing at least a portion of a data processing pipeline, the data processing pipeline including a sequence of data processing operations, the first master node and the first worker node comprising a distributed cluster of computing nodes that further includes a second worker node, and the one or more trace messages included in the first trace stream corresponding to one or more events occurring during the execution of at least the portion of the data processing pipeline; 
 in response to the request from the client, subscribing, by the first master node, to the first trace stream published by the first worker node, the first master node subscribing, based at least on the client requesting to receive the first trace stream published by the first worker node but not a second trace stream published by the second worker node, to the first trace stream published by the first worker node but not the second trace stream published by the second worker node; 
 in response to a failure at the first master node, subscribing, by a second master node, to the first trace stream published by the first worker node; and 
 generating, by the first master node or the second master node, a user interface for displaying, at the client, the one or more trace messages included in the first trace stream published by the first worker node. 
 
     
     
       10. The method of  claim 9 , wherein the first worker node executes a first portion of the data processing pipeline and the second worker node executes a second portion of the data processing pipeline, and wherein the first master node coordinates the execution of the data processing pipeline by the first worker node and the second worker node. 
     
     
       11. The method of  claim 10 , further comprising:
 receiving, from the client, another request to receive a second trace stream including one or more trace messages output by the second worker node during the execution of the second portion of the data processing pipeline; and 
 responding to the other request by at least subscribing to the second trace stream published by the second worker node. 
 
     
     
       12. The method of  claim 11 , further comprising:
 storing, in a trace log, at least one of the one or more trace messages included in the first trace stream published by the first worker node and the one or more trace messages included in the second trace stream published by the second worker node. 
 
     
     
       13. The method of  claim 10 , wherein the first portion of the data processing pipeline and the second portion of the data processing pipeline each include one or more data processing operations from the sequence of data processing operations, and wherein the one or more data processing operations are performed on data stored in a database. 
     
     
       14. The method of  claim 9 , wherein a listener is deployed at the first worker node in response to the first master node subscribing to the first trace stream published by the first worker node, wherein the listener is configured to detect when the first worker node outputs a trace message, and wherein at least some trace messages output by the first worker node are pushed to the first master node. 
     
     
       15. The method of  claim 14 , wherein the one or more trace messages output by the first worker node are filtered prior to being pushed to the first master node, and wherein the one or more trace messages are filtered based at least on a threshold severity level specified by the client. 
     
     
       16. A non-transitory computer-readable medium storing instructions, which when executed by at least one data processor, result in operations comprising:
 receiving, at a first master node, a request from a client to receive a first stream including one or more trace messages output by a first worker node executing at least a portion of a data processing pipeline, the data processing pipeline including a sequence of data processing operations, the first master node and the first worker node comprising a distributed cluster of computing nodes that further includes a second worker node, and the one or more trace messages included in the first trace stream corresponding to one or more events occurring during the execution of at least the portion of the data processing pipeline; 
 in response to the request from the client, subscribing, by the first master node, to the first trace stream published by the first worker node, the first master node subscribing, based at least on the client requesting to receive the first trace stream published by the first worker node but not a second trace stream published by the second worker node, to the first trace stream published by the first worker node but not the second trace stream published by the second worker node; 
 in response to a failure at the first master node, subscribing, by a second master node, to the first trace stream published by the first worker node; and 
 generating, by the first master node or the second master node, a user interface for displaying, at the client, the one or more trace messages included in the first trace stream published by the first worker node.