Continuous anomaly detection based on behavior modeling and heterogeneous information analysis

Abstract

The present disclosure describes a continuous anomaly detection method and system based on multi-dimensional behavior modeling and heterogeneous information analysis. A method includes collecting data, processing and categorizing a plurality of events, continuously clustering the plurality of events, continuously model building for behavior and information analysis, analyzing behavior and information based on a holistic model, detecting anomalies in the data, displaying an animated and interactive visualization of a behavioral model, and displaying an animated and interactive visualization of the detected anomalies.

1 . A method of continuous anomaly detection based on behavioral modeling and heterogeneous information analysis, the method comprising:
 collecting data;   processing and categorizing a plurality of events from the data;   continuously clustering the plurality of events;   continuously model building for behavior and information analysis;   analyzing behavior and information based on a holistic model;   detecting anomalies in the data;   displaying an animated and interactive visualization of a behavioral model; and   displaying an animated and interactive visualization of the detected anomalies.   
     
     
         2 . The method of  claim 1 , wherein a flow of an incoming data stream and an aging of data from the data stream are governed by one or more event scoping policies, which can be configured according to a strategy including one of a sliding window strategy or a least-relevant-first strategy. 
     
     
         3 . The method of  claim 1 , wherein data is collected from a data source, in an online or offline manner, and either incrementally or continuously. 
     
     
         4 . The method of  claim 3 , wherein a collection mode and a collection rate of each data source are continuously adjusted based on parameters selected from the group consisting of data collection throughput, data processing throughput, data prioritization, data lifecycle, and data relevance for anomaly detection. 
     
     
         5 . The method of  claim 3 , wherein new features of the data are automatically discovered during data collection. 
     
     
         6 . The method of  claim 1 , wherein an incoming stream of events is clustered in real time based on a similarity measure between events, producing usable sets of clusters and updating them as events are acquired from the underlying data stream. 
     
     
         7 . The method of  claim 6 , further comprising querying for current results or a current state of a clustering process. 
     
     
         8 . The method of  claim 6 , wherein clustering on the same set of events streamed in different orders produces the same result. 
     
     
         9 . The method of  claim 1 , further comprising continuously deriving evidence from an incoming stream of events. 
     
     
         10 . The method of  claim 9 , wherein the evidence is stored as a hypergraph including a representation of nodes and edges that satisfies the property of monotonicity for evidence accrual, and wherein discussions are built from the evidence in real time using criteria to close pending discussions. 
     
     
         11 . (canceled) 
     
     
         12 . The method of  claim 1 , further comprising running a number of queries on an incoming stream of events by compiling queries into a Petri net structure. 
     
     
         13 . The method of  claim 1 , further comprising automatically mining periodic events in the presence of irregularities, wherein the periodic events are presented in a continuous visualization, and correlations can be established across periodic events, and wherein the mining of periodic events includes real-time indexing and querying of a periodic patterns database based on structural information or semantic information. 
     
     
         14 . (canceled) 
     
     
         15 . The method of  claim 1 , wherein different types of anomalies are detected based on the results of data analysis and individual and collective behavior modeling. 
     
     
         16 . The method of  claim 15 ,
 wherein an external monitoring or anomaly detection system provides a source of anomalies,   wherein baseline patterns are computed with respect to different referentials and used as a source of anomalies corresponding to deviations from the baseline patterns, and   wherein rankings of individuals against behavioral traits are a source of anomalies corresponding to abnormal behavior.   
     
     
         17 . (canceled) 
     
     
         18 . (canceled) 
     
     
         19 . The method of  claim 1 , further comprising:
 detecting anomalies with third-party systems such as rule-based anomaly detection systems; and   generating actionable alerts by aggregating anomalies along time windows, across individuals or groups, and/or across an unbounded number of dimensions in the data.   
     
     
         20 . (canceled) 
     
     
         21 . The method of  claim 1 , wherein detecting anomalies is tuned based on user feedback which is maintained and adjusted automatically over time. 
     
     
         22 . The method of  claim 1 , wherein the results of anomaly detection are presented in a user interface in which anomalies can be browsed by detection time or by behavioral trait, and in which the user can efficiently give feedback on the results of anomaly detection to ensure sustained accuracy and performance of anomaly detection. 
     
     
         23 . The method of  claim 1 , further comprising automatically detecting workflow models and instances of workflow models from the incoming stream of events. 
     
     
         24 . (canceled) 
     
     
         25 . (canceled) 
     
     
         26 . (canceled) 
     
     
         27 . (canceled) 
     
     
         28 . (canceled) 
     
     
         29 . (canceled) 
     
     
         30 . (canceled) 
     
     
         31 . (canceled) 
     
     
         32 . (canceled) 
     
     
         33 . (canceled) 
     
     
         34 . (canceled) 
     
     
         35 . (canceled) 
     
     
         36 . (canceled) 
     
     
         37 . (canceled) 
     
     
         38 . (canceled) 
     
     
         39 . (canceled) 
     
     
         40 . (canceled) 
     
     
         41 . (canceled) 
     
     
         42 . (canceled) 
     
     
         43 . (canceled) 
     
     
         44 . (canceled) 
     
     
         45 . A method for detecting textblock patterns, the method comprising:
 iterating over a universe of items;   tokenizing text of each item;   forming n-grams from a stream of tokens;   sliding a window of size k over successive n-grams;   producing a directed weighted graph of transitions between the n-grams co-occurring in the sliding window, such that the graph of transitions is accumulated over all items in the universe of items;   calculating a local clusterability of each vertex in the graph of transitions;   using the calculated local clusterability of each vertex to determine whether to keep or discard the vertex in question;   detecting and labeling connected components amongst the kept vertices; and   identifying the connected components with textblock patterns.   
     
     
         46 . (canceled) 
     
     
         47 . (canceled) 
     
     
         48 . (canceled) 
     
     
         49 . A method for finding textblock hits within an item, comprising:
 tokenizing text of the item to form tokens;   forming n-grams from the tokens;   running a sliding window of size k over the n-grams;   looking up transitions between n-grams co-occurring in the window in the graph of textblock patterns; and   examining close runs of transitions to determine if the close runs of transitions constitute textblock hits based on recall and precision calculations.   
     
     
         50 . (canceled) 
     
     
         51 . (canceled) 
     
     
         52 . (canceled) 
     
     
         53 . (canceled) 
     
     
         54 . (canceled) 
     
     
         55 . (canceled) 
     
     
         56 . (canceled) 
     
     
         57 . (canceled) 
     
     
         58 . (canceled) 
     
     
         59 . (canceled) 
     
     
         60 . (canceled) 
     
     
         61 . (canceled) 
     
     
         62 . (canceled) 
     
     
         63 . (canceled)