System and method of advanced event ranking and correlation for threat detection

Abstract

Systems and methods for advanced event ranking and correlation for threat detection. A method includes real-time processing of events with stateful threat detection, combining immediate detection capabilities with sophisticated threat analysis. A method further includes multi-stage processing in a detection engine, where generic events from endpoint detection and response (EDR) agents are scored and enriched to provide extended context based on machine learning models for event scoring, enriched events correlation, and applying security rules to detect threats.

1 . A method for adaptive threat detection in a computing system comprising a microprocessor, memory, and a plurality of endpoints, the method comprising:
 collecting events on the endpoints and sending the events to an event router;   processing the events on the event router under program control of the microprocessor in the form of a persistent event stream, wherein the processing includes:
 normalizing the events and assigning topics to the events to prioritize event enrichment operations and subsequent detection engine operations based on event scores, wherein normalizing the events further comprises converting the events into a standardized event format; 
   scoring the events from the persistent event stream using at least one of a lookup tables, a serialized machine learning model or a baselining machine learning model, wherein each event is scored to indicate a likelihood that the event represents a security threat;   enriching generic events at an event enrichment unit using at least one of the lookup tables or the baselining machine learning model, wherein the enriching includes incorporating at least one additional context for a given generic event, and wherein the enriching is prioritized according to the event score and at least one topic of the generic event to create enriched scored events;   publishing, by the event enrichment unit, the enriched scored events to the persistent event stream by joining standardized event data with the at least one additional context, wherein the persistent event stream is at least partially organized by topics;   processing the enriched scored events from the persistent event stream in a detection engine, wherein the processing includes:
 detecting a first detection in the persistent event stream within a short-term time window, the first detection comprising an immediate potential threat, 
 tagging the first detection with a temporal marker for association in an event correlation lifetime, 
 retrieving other events in the persistent event stream, 
 correlating the registered first detection with the other events in the persistent event stream within a long-term time window according to the event correlation lifetime, wherein the registered first detection has a relationship to the other events by at least one of the standardized event data or the at least one additional context, and 
 detecting a second detection using the correlating, wherein the second detection comprises a more complex potential threat with higher severity than the first detection; and 
   registering a security incident when the second detection has a severity higher than a predefined threshold;   wherein the baselining machine learning model, the serialized machine learning model, the lookup table, and the machine learning-based detection engine use the standardized event format.   
     
     
         2 . The method of  claim 1 , wherein the events collected on endpoints include process start, file access, network connections, and registry changes. 
     
     
         3 . The method of  claim 1 , wherein the topics assigned to events include event attribute source, type, score or severity. 
     
     
         4 . The method of  claim 1 , wherein the lookup tables used for scoring are generated based on historical event data stored in an event database configured to collect all events processed in the persistent event stream. 
     
     
         5 . The method of  claim 1 , wherein the baselining machine learning model is trained using historical event data to recognize normal behavior patterns and detect deviations. 
     
     
         6 . The method of  claim 1 , wherein the event enrichment unit uses additional data sources including third-party threat intelligence feeds, system logs, network traffic data, and user behavior analytics for event enrichment. 
     
     
         7 . The method of  claim 1 , wherein the first detection is registered based on predefined security rules applied to the enriched scored events. 
     
     
         8 . The method of  claim 1 , wherein the second time window for correlating events is dynamically adjusted based on the severity of the first engine detection. 
     
     
         9 . The method of  claim 1 , wherein the second detection of higher severity is based on the aggregation of multiple correlated events and correlated events scores. 
     
     
         10 . The method of  claim 1 , wherein the event enrichment unit prioritizes the enrichment of events based on event scores and related event topics. 
     
     
         11 . A system for adaptive threat detection in a computing system, the system comprising:
 an endpoint detection and response (EDR) agent configured to collect events on an endpoint and transfer the events to an event router;   at least one processor and memory operably coupled to the at least one processor;   instructions that, when executed by the at least one processor, cause the at least one processor to implement:
 the event router, the event router configured to process the events in the form of a persistent event stream, wherein the processing includes:
 normalizing the events and assigning topics to the events to control the event enrichment pipeline and prioritize subsequent operations based on event scores, wherein normalizing the events further comprises converting the events into a standardized event format; 
 
 an event scoring unit configured to score the events from the persistent event stream using lookup tables and infer a security risk score for each event, wherein the scoring is performed using serialized machine learning models and baselining machine learning models, wherein each event is scored to indicate a likelihood that the event represents a security threat; 
 an event enrichment unit configured to:
 enrich generic events using the baselining machine learning model, wherein the enriching includes incorporating at least one additional context for a given generic event, and wherein the enrichment is performed in a prioritized manner according to the event score and at least one topic of the generic event to create enriched scored events, and 
 publish the enriched scored events to the persistent event stream by joining standardized event data with the at least one additional context, wherein the persistent event stream is at least partially organized by topics; 
 
 a detection engine configured to process the enriched scored events from the persistent event stream, wherein the processing includes:
 detecting a first detection in the persistent event stream within a short-term time window, the first detection comprising an immediate potential threat, 
 tagging the first detection with a temporal marker for association in an event correlation lifetime, 
 retrieving other events in the persistent event stream, 
 correlating the registered first engine detection with the other events in the persistent event stream over a long-term time window according to the event correlation lifetime, wherein the registered first detection has a relationship to the other events by at least one of the standardized event data or the at least one additional context, 
 detecting a second detection using the correlating, wherein the second detection comprises a more complex potential threat with higher severity than the first engine detection, and 
 registering a security incident when the second detection has a severity higher than a predefined threshold; 
 
 wherein the baselining machine learning model, serialized machine learning model, lookup table, and machine learning-based detection engine use the standardized event format. 
   
     
     
         12 . The system of  claim 11 , wherein the EDR agent is configured to collect events including process start, file access, network connections, and registry changes. 
     
     
         13 . The system of  claim 11 , wherein the topics assigned to events include event attribute source, type, score or severity. 
     
     
         14 . The system of  claim 11 , wherein the lookup tables used for scoring are generated based on historical event data stored in an event database configured to collect all events processed in the persistent event stream. 
     
     
         15 . The system of  claim 11 , wherein the baselining machine learning models are trained using historical event data to recognize normal behavior patterns and detect deviations. 
     
     
         16 . The system of  claim 11 , wherein the event enrichment unit uses additional data sources including third-party threat intelligence feeds, system logs, network traffic data, and user behavior analytics for event enrichment. 
     
     
         17 . The system of  claim 11 , wherein the first detection is registered based on predefined security rules applied to the enriched scored events. 
     
     
         18 . The system of  claim 11 , wherein the second time window for correlating events is dynamically adjusted based on the severity of the first engine detection. 
     
     
         19 . The system of  claim 11 , wherein the second detection of higher severity is based on the aggregation of multiple correlated events and correlated events scores. 
     
     
         20 . The system of  claim 11 , wherein the event enrichment unit prioritizes the enrichment of events based on event scores and related event topics.