Real-time observability and enhanced resiliency for event processing pipelines

Abstract

Mechanisms are provided for generating a holistic view of an event processing pipeline of an event driven architecture. Curation of runtime information from components of an event processing pipeline is performed, where the runtime information comprises real-time runtime metrics, aggregated logs, event tracing/recording data structures, and configuration information. The curated runtime information is correlated to generate correlated runtime data. Predictive models and predictive analytics process the correlated runtime data to predict anomalies in the components of the event processing pipeline. A fault circuit interrupt machine (FCIM) processes the predicted anomalies based on user resiliency configurations to determine recommended event resiliency remediation actions for addressing the predicted anomalies. A real-time holistic view of the components of the event processing pipeline is generated based on the correlated runtime data, the predicted anomalies, and the determined recommended event resiliency remediation actions.

What is claimed is: 
     
         1 . A method, in a data processing system, for generating a holistic view of an event processing pipeline of an event driven architecture, the method comprising:
 performing curation of runtime information from components of an event processing pipeline, wherein the runtime information comprises real-time runtime metrics, aggregated logs of the components of the event processing pipeline, event tracing data structures, and configuration information of the components of the event processing pipeline;   correlating the real-time runtime metrics, aggregated logs, and event tracing data structures of the curated runtime information to generate correlated runtime data;   processing, by predictive models and predictive analytics, the correlated runtime data to predict one or more anomalies in the components of the event processing pipeline;   processing, by a fault circuit interrupt machine (FCIM), the predicted one or more anomalies based on a user resiliency configuration to determine one or more recommended event resiliency remediation actions for addressing the predicted one or more anomalies; and   generating and outputting a graphical user interface comprising a real-time holistic view of the components of the event processing pipeline based on the correlated runtime data, the predicted one or more anomalies, and the determined one or more recommended event resiliency remediation actions.   
     
     
         2 . The method of  claim 1 , wherein one or more elements of the real-time holistic view, corresponding to the components of the event processing pipeline, are rendered with attributes that distinguish the elements from one another based on the type of predicted anomalies, and severity of the predicted anomalies, associated with the components corresponding to the elements. 
     
     
         3 . The method of  claim 1 , wherein the user resiliency configuration maps one or more specific use cases, corresponding to attributes of the event processing pipeline, with possible predicted anomalies, and specifies one or more of the event resiliency remediation actions corresponding to each of the one or more specific use cases and possible predicted anomalies. 
     
     
         4 . The method of  claim 1 , wherein processing the correlated runtime data to predict anomalies in the components of the event processing pipeline comprises:
 identifying an event outlier based on at least one of performance indicators or success rate key performance indicators;   identifying, from the runtime information, telemetry data corresponding to the event outlier and logs of components of the event processing pipeline that are involved in event processing of the event outlier;   analyzing the telemetry data and logs to identify metadata attributes associated with the event outlier;   executing an associativity scoring model on the identified metadata attributes to rank and order the metadata attributes based on a propensity to cause failures, to thereby generate a finalized ranked listing of metadata attributes; and   predicting anomalies based on the finalized ranked listing of metadata attributes.   
     
     
         5 . The method of  claim 4 , wherein executing the associativity scoring model further comprises filtering out metadata attributes that have a rank below a predetermined threshold rank. 
     
     
         6 . The method of  claim 4 , wherein predicting anomalies based on the finalized ranked listing of metadata attributes comprises processing the finalized ranked listing of metadata attributes, along with component health metrics for the components of the event processing pipeline, as input to a trained machine learning computer model that predicts one or more of a slowdown buildup in the event processing pipeline, a failure of a component in the event processing pipeline, or a root cause of an anomaly in the event processing pipeline. 
     
     
         7 . The method of  claim 1 , wherein the FCIM causes the recommended event resiliency remediation action to be executed by at least one of moving one or more event messages to a functionally compliant alternative event processing pipeline based on a type of identified event metadata, selectively failing event message processing of selected event messages based on a specific identified event metadata of event messages causing slowdown or failures in the event processing pipeline, or selectively offload event messages for offline batch processing based on specific identified event metadata of event messages causing slowdown or failures in the event processing pipeline. 
     
     
         8 . The method of  claim 1 , wherein the real-time holistic view of the components of the event processing pipeline is updated with annotations or modifications to textual or graphical attributes in real-time as new runtime information is curated and correlated. 
     
     
         9 . The method of  claim 1 , wherein the predicted anomalies are predicted slowdowns in processing of messages through the event processing pipeline or predicted failures of a component of the event processing pipeline. 
     
     
         10 . The method of  claim 1 , wherein the real-time holistic view of the components of the event processing pipeline comprise annotations specifying types of anomalies associated with elements of the real-time holistic view and probable root causes of the anomalies associated with the elements of the real-time holistic view. 
     
     
         11 . A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:
 perform curation of runtime information from components of an event processing pipeline, wherein the runtime information comprises real-time runtime metrics, aggregated logs of the components of the event processing pipeline, event tracing data structures, and configuration information of the components of the event processing pipeline;   correlate the real-time runtime metrics, aggregated logs, and event tracing data structures of the curated runtime information to generate correlated runtime data;   process, by predictive models and predictive analytics, the correlated runtime data to predict one or more anomalies in the components of the event processing pipeline;   process, by a fault circuit interrupt machine (FCIM), the predicted one or more anomalies based on a user resiliency configuration to determine one or more recommended event resiliency remediation actions for addressing the predicted one or more anomalies; and   generate and output a graphical user interface comprising a real-time holistic view of the components of the event processing pipeline based on the correlated runtime data, the predicted one or more anomalies, and the determined one or more recommended event resiliency remediation actions.   
     
     
         12 . The computer program product of  claim 11 , wherein one or more elements of the real-time holistic view, corresponding to the components of the event processing pipeline, are rendered with attributes that distinguish the elements from one another based on the type of predicted anomalies, and severity of the predicted anomalies, associated with the components corresponding to the elements. 
     
     
         13 . The computer program product of  claim 11 , wherein the user resiliency configuration maps one or more specific use cases, corresponding to attributes of the event processing pipeline, with possible predicted anomalies, and specifies one or more of the event resiliency remediation actions corresponding to each of the one or more specific use cases and possible predicted anomalies. 
     
     
         14 . The computer program product of  claim 11 , wherein the computer readable program further causes the computing device to process the correlated runtime data to predict anomalies in the components of the event processing pipeline at least by:
 identifying an event outlier based on at least one of performance indicators or success rate key performance indicators;   identifying, from the runtime information, telemetry data corresponding to the event outlier and logs of components of the event processing pipeline that are involved in event processing of the event outlier;   analyzing the telemetry data and logs to identify metadata attributes associated with the event outlier;   executing an associativity scoring model on the identified metadata attributes to rank and order the metadata attributes based on a propensity to cause failures, to thereby generate a finalized ranked listing of metadata attributes; and   predicting anomalies based on the finalized ranked listing of metadata attributes.   
     
     
         15 . The computer program product of  claim 14 , wherein executing the associativity scoring model further comprises filtering out metadata attributes that have a rank below a predetermined threshold rank. 
     
     
         16 . The computer program product of  claim 14 , wherein predicting anomalies based on the finalized ranked listing of metadata attributes comprises processing the finalized ranked listing of metadata attributes, along with component health metrics for the components of the event processing pipeline, as input to a trained machine learning computer model that predicts one or more of a slowdown buildup in the event processing pipeline, a failure of a component in the event processing pipeline, or a root cause of an anomaly in the event processing pipeline. 
     
     
         17 . The computer program product of  claim 11 , wherein the FCIM causes the recommended event resiliency remediation action to be executed by at least one of moving one or more event messages to a functionally compliant alternative event processing pipeline based on a type of identified event metadata, selectively failing event message processing of selected event messages based on a specific identified event metadata of event messages causing slowdown or failures in the event processing pipeline, or selectively offload event messages for offline batch processing based on specific identified event metadata of event messages causing slowdown or failures in the event processing pipeline. 
     
     
         18 . The computer program product of  claim 11 , wherein the real-time holistic view of the components of the event processing pipeline is updated with annotations or modifications to textual or graphical attributes in real-time as new runtime information is curated and correlated. 
     
     
         19 . The computer program product of  claim 11 , wherein the predicted anomalies are predicted slowdowns in processing of messages through the event processing pipeline or predicted failures of a component of the event processing pipeline. 
     
     
         20 . An apparatus comprising:
 at least one processor; and   at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to:   perform curation of runtime information from components of an event processing pipeline, wherein the runtime information comprises real-time runtime metrics, aggregated logs of the components of the event processing pipeline, event tracing data structures, and configuration information of the components of the event processing pipeline;   correlate the real-time runtime metrics, aggregated logs, and event tracing data structures of the curated runtime information to generate correlated runtime data;   process, by predictive models and predictive analytics, the correlated runtime data to predict one or more anomalies in the components of the event processing pipeline;   process, by a fault circuit interrupt machine (FCIM), the predicted one or more anomalies based on a user resiliency configuration to determine one or more recommended event resiliency remediation actions for addressing the predicted one or more anomalies; and   generate and output a graphical user interface a real-time holistic view of the components of the event processing pipeline based on the correlated runtime data, the predicted one or more anomalies, and the determined one or more recommended event resiliency remediation actions.