Methods and Systems for Exception Detection and Reporting in a Distributed Network System

Abstract

In one embodiment, the methods and systems may include receiving an Application Program Interface (API) request, assigning a unique identifier to the API request, recording, in association with the unique identifier, receipt time corresponding to a time at which the API request is received, and associating one or more system state changes with the unique identifier. Advantageously, the present embodiments provide a more accurate failure monitoring and associated metrics. Additional benefits and advantages of the present embodiments will become evident in the following description.

1 - 22 . (canceled) 
     
     
         1 . A system for exception detection and reporting in a distributed system, the system comprising:
 a first processing element, a second processing element, and a third processing element, each processing element being operably coupled with a computer readable medium, memory, processor, and network connection, and including one or more functions, each function comprising code interpretable by the processor to accept a set of inputs and produce an output;   wherein each of the first, second, and third processing elements are communicably coupled to each other according to an internal processing graph, the processing graph including a external call Application Programming Interface (API) endpoint, and internal element output to element input connections;   a compute controller separate from the first, second, and third processing elements, the compute controller including a set of discrete, communicably coupled interoperating components, the set of interoperating components including one or more instrumentation components operably connected to each of the first, second, and third processing elements, the compute controller operable to insert instrumentation components into the processing graph according to a changeable configuration;   wherein a instrumentation component associated with the first processing element sets at least one trace on the first processing element, the trace type being selected from a group consisting of a recorder observing and correlating inputs and outputs from the first processing element, an interceptor modifying one or more functions in the first processing element, and an event reporter;   wherein a instrumentation component associated with the second processing element sets at least one trace on the second processing element, the trace type being selected from a group consisting of a recorder observing and correlating inputs and outputs from the second processing element, an interceptor modifying one or more functions in the second processing element, and an event reporter;   wherein an instrumentation component associated with the third processing element sets at least one trace on the third processing element, the trace type being selected from a group consisting of a recorder observing and correlating inputs and outputs from the third processing element, an interceptor modifying one or more functions in the third processing element, and an event reporter;   wherein each instrumentation component is operable to communicate exceptional condition messages to the controller, exceptional conditions including errors, unexpected results, utilization of processing, storage, or memory capacity above a threshold defined by the configuration, or a response latency above a threshold defined by the configuration;   wherein the controller is further operable to evaluate received exceptional condition messages and take an action selected from a group consisting of writing an exception report to a log, canceling an in-process computation moving through the processing graph, and reporting an exceptional condition via the API endpoint.   
     
     
         2 . The system of  claim 1 , wherein at least two of the processing elements are operably coupled to a shared computer readable medium, memory, processor, and network connection. 
     
     
         3 . The system of  claim 1 , wherein at least two of the processing elements communicate via a process-internal call graph, inter-process communication (IPC), or a shared memory communication system. 
     
     
         4 . The system of  claim 1 , wherein each instrumentation component is additionally operable to communicate non-exceptional messages, and the controller is additionally operable to write non-exceptional reports to a log. 
     
     
         5 . The system of  claim 1 , wherein an instrumentation component is additionally operable to communicate non-exceptional messages, and the controller is additionally operable to write non-exceptional reports to a log. 
     
     
         6 . The system of  claim 1 , further comprising a generator operable to create an identifier;
 wherein an instrumentation component associated with each processing element is further operable to associate each set of computations associated with a unique movement of inputs and outputs through the processing graph with an identifier created by the generator, wherein the portion of the instrumentation component associating part of the set of computations includes code that, when executed:   at processing element associated with the API endpoint, modifies the output of the associated function to include the identifier;   at each successive function invocation point along the computation graph, receives inputs including the identifier, temporarily stores the identifier, and passes the received inputs, without the identifier, to the traced function, and upon receiving the outputs of the functions, modifies the output of the function to include the identifier.   
     
     
         7 . The system of  claim 6 , wherein an instrumentation component is additionally operable to communicate non-exceptional messages, and the controller is additionally operable to write non-exceptional reports to a log. 
     
     
         8 . The system of  claim 7 , wherein one type of non-exceptional message includes the time needed to traverse one portion of the processing graph. 
     
     
         9 . The system of  claim 7 , wherein one type of non-exceptional message includes the time needed to traverse the entire processing graph. 
     
     
         10 . A system for exception detection and reporting in a distributed system, the system comprising:
 a set of processing elements, including at least a first processing element, a second processing element, and a third processing element, each processing element being operably coupled with a computer readable medium, memory, processor, and network connection, and including one or more functions, each function comprising code interpretable by the processor to accept a set of inputs and produce an output;   a instrumentation component comprising code, that when executed on one or more of the processing elements:
 loads a configuration identifying one or more target functions to trace, the location of the functions being determined relative to the processing element being used to execute the function in the context of a processing graph, the processing graph including an external application programming interface (API) endpoint; 
   patches each target function identified in the configuration, wherein the patch type is selected from a group consisting of a recorder patch observing and correlating inputs and outputs to the patched target function, an interceptor patch modifying the inputs and outputs accepted or produced by the function from their original state, and an event reporter patch operable to identify and communicate changes in the state of the portion of the system observed;   a controller component including code, that when executed:
 executes the patched target function on the processing element processor; 
   a monitor component communicably coupled to the processing elements executing patched functions, the monitor component including code, that when executed:
 identifies exceptional conditions including errors, unexpected results, utilization of processing, storage, or memory capacity above a threshold defined by the configuration, or a response latency above a threshold defined by the configuration, and non-exceptional conditions including utilization of a resource and time elapsed during execution of the function; and 
 takes an action selected from a group consisting of writing a report to a log, canceling an in-process computation moving through the processing graph, reporting a condition via the API endpoint, and reporting a condition to a second API endpoint. 
   
     
     
         11 . The system of  claim 10 , wherein at least two of the processing elements are operably coupled to a shared computer readable medium, memory, processor, and network connection. 
     
     
         12 . The system of  claim 10 , wherein at least two of the processing elements communicate via a process-internal call graph, inter-process communication (IPC), or a shared memory communication system. 
     
     
         13 . The system of  claim 10 , wherein each instrumentation component additionally includes to communicate non-exceptional messages, and the controller is additionally operable to write non-exceptional reports to a log. 
     
     
         14 . The system of  claim 10 , further comprising a generator operable to create an identifier;
 wherein an instrumentation component associated with each processing element is further operable to associate each set of computations associated with a unique movement of inputs and outputs through the processing graph with an identifier created by the generator, wherein the portion of the instrumentation component associating part of the set of computations includes code that, when executed:   at processing element associated with the API endpoint, modifies the output of the associated function to include the identifier;   at each successive function invocation point along the computation graph, receives inputs including the identifier, temporarily stores the identifier, and passes the received inputs, without the identifier, to the traced function, and upon receiving the outputs of the functions, modifies the output of the function to include the identifier.