System and method for real-time detection of anomalies in database usage

Abstract

ABSTRACT A system and method for real-time detection of anomalies in database or application usage is disclosed. Embodiments provide a mechanism to detect anomalies in database or application usage, such as data exfiltration attempts, first by identifying correlations (e.g., patterns of normalcy) in events across different heterogeneous data streams (such as those associated with ordinary, authorized and benign database usage, workstation usage, user behavior or application usage) and second by identifying deviations/anomalies from these patterns of normalcy across data streams in real-time as data is being accessed. An alert is issued upon detection of an anomaly, wherein a type of alert is determined based on a characteristic of the detected anomaly.

What is claimed is: 
     
         1 . A method for real-time detection of anomalies occurring in an enterprise computer network, comprising:
 receiving a plurality of heterogeneous data streams from sources in the network, the sources including two levels, first level sources and second level sources,   wherein the first level sources include one or more selected from a group consisting of agents located at databases, agents located at applications, audit programs located at user workstations, sensors located in the network, and sensors located at access points to the network,   wherein the second level sources include one or more selected from a group consisting of data access, user behavior, computer activity and network activity, and   wherein the first level sources monitor event streams of the second level sources and generate data streams indicative of corresponding second level source activity in a uniform format;   processing the heterogeneous data streams obtained by combining at least two of the first level sources to identify events therein, each event being identified by at least a unique ID, a timestamp, and an event type, wherein the processing of the heterogeneous data streams includes combining at least two of the first level sources into a single data stream;   correlating the processed heterogeneous data streams to form an integrated data stream comprising a plurality of identified events;   detecting the existence and at least one characteristic of an anomaly in the computer network by application of a predetermined model of normalcy and one or more anomaly rules to the integrated data stream comprising the plurality of identified events; and   issuing an alert based on the at least one characteristic of the anomaly.   
     
     
         2 . The method of  claim 1  further comprising creating the predetermined model of normalcy and the one or more anomaly rules by:
 receiving additional data comprising the plurality of heterogeneous data streams, wherein the additional data corresponds to authorized and benign usage of network resources; 
 processing the heterogeneous data streams to identify events therein, each even being identified by at least a unique ID, a timestamp, and an event type; 
 correlating the processed data streams to form an integrated data stream comprising a plurality of identified events; 
 identifying one or more patterns from relations between identified events comprising the integrated data stream; and 
 creating the model of normalcy and the one or more anomaly rules based on the identified one or more patterns. 
 
     
     
         3 . The method of  claim 1  wherein the one or more anomaly rules relate to at least one of how and whether anomalies are detected, how a detected anomaly is treated and characterized, and what reaction to employ in response to the detected anomaly. 
     
     
         4 . The method of  claim 1  further comprising:
 estimating a number or frequency of one or more event types in the processed data stream without searching the entire processed data stream; and 
 determining one or more temporal, spatial, or generalized associations between a plurality of events in the processed data stream. 
 
     
     
         5 . The method of  claim 1  wherein the detected anomaly is indicative of unauthorized manipulation or falsification of data, sabotage of a database, or exfiltration of data. 
     
     
         6 . The method of  claim 1  wherein the heterogeneous data streams comprise multi-modal asynchronous signals. 
     
     
         7 . The method of  claim 1  wherein the program code includes an algorithm that detects and extracts persistent events among the plurality of identified events in at least one of the plurality of heterogeneous data streams, and wherein the persistent events are time-stamped data that appear regularly over time. 
     
     
         8 . The method of  claim 7  wherein the persistent events appear in different distributed streams among the plurality of heterogeneous data streams. 
     
     
         9 . The method of  claim 7  wherein the at least one of the plurality of heterogeneous data streams is statistically sampled to reduce stream size of the at least one of the plurality of heterogeneous data streams, without overlooking the persistent events. 
     
     
         10 . A method for real-time detection of anomalies occurring in an enterprise computer network, comprising:
 receiving a plurality of heterogeneous data streams from sources in the network, the sources including two levels, first level sources and second level sources,   wherein the first level sources include one or more selected from a group consisting of agents located at databases; agents located at applications; audit programs located at user workstations; sensors located in the network; and sensors located at access points to the network,   wherein the second level sources include one or more selected from a group consisting of data access, user behavior, computer activity and network activity, and wherein the first level sources monitor event streams of the second level sources and generate data streams indicative of corresponding second level source activity in a uniform format;   processing the heterogeneous data streams obtained by combining at least two of the first level sources to identify events therein, each event being identified by at least a unique ID, a timestamp, and an event type, wherein the processing of the heterogeneous data streams includes:
 combining at least two of the first level sources into a single data stream; and 
 operating on the single data stream using an algorithm that identifies spatiotemporal relationships; 
   correlating the processed heterogeneous data streams to form an integrated data stream comprising a plurality of identified events;   detecting the existence and at least one characteristic of an anomaly in the computer network by application of a predetermined model of normalcy and one or more anomaly rules to the integrated data stream comprising the plurality of identified events; and   issuing an alert based on the at least one characteristic of the anomaly.   
     
     
         11 . The method of  claim 10  further comprising creating the predetermined model of normalcy and the one or more anomaly rules by:
 receiving additional data comprising the plurality of heterogeneous data streams, wherein the additional data corresponds to authorized and benign usage of network resources; 
 processing the heterogeneous data streams to identify events therein, each even being identified by at least a unique ID, a timestamp, and an event type; 
 correlating the processed data streams to form an integrated data stream comprising a plurality of identified events; 
 identifying one or more patterns from relations between identified events comprising the integrated data stream; and 
 creating the model of normalcy and the one or more anomaly rules based on the identified one or more patterns. 
 
     
     
         12 . The method of  claim 10  wherein the one or more anomaly rules relate to at least one of how and whether anomalies are detected, how a detected anomaly is treated and characterized, and what reaction to employ in response to the detected anomaly. 
     
     
         13 . The method of  claim 10  further comprising:
 estimating a number or frequency of one or more event types in the processed data stream without searching the entire processed data stream; and 
 determining one or more temporal, spatial, or generalized associations between a plurality of events in the processed data stream. 
 
     
     
         14 . The method of  claim 10  wherein the detected anomaly is indicative of unauthorized manipulation or falsification of data, sabotage of a database, or exfiltration of data. 
     
     
         15 . The method of  claim 10  wherein the heterogeneous data streams comprise multi-modal asynchronous signals. 
     
     
         16 . The method of  claim 10  wherein the program code includes an algorithm that detects and extracts persistent events among the plurality of identified events in at least one of the plurality of heterogeneous data streams, and wherein the persistent events are time-stamped data that appear regularly over time. 
     
     
         17 . The method of  claim 16  wherein the persistent events appear in different distributed streams among the plurality of heterogeneous data streams. 
     
     
         18 . The method of  claim 16  wherein the at least one of the plurality of heterogeneous data streams is statistically sampled to reduce stream size of the at least one of the plurality of heterogeneous data streams, without overlooking the persistent events. 
     
     
         19 . A method for real-time detection of anomalies occurring in a computer network, comprising:
 receiving a plurality of heterogeneous data streams from sources in the network, the sources including first level sources and second level sources, wherein the first level sources include one or more selected from a group consisting of agents located at databases, agents located at applications, audit programs located at user workstations, sensors located in the network, and sensors located at access points to the network; wherein the second level sources include event streams to be analyzed,   wherein the first level sources monitor the event streams of the second level sources and generate data streams indicative of corresponding second level source activity in a uniform format, and   wherein each of the heterogeneous data streams is obtained by combining at least two of the first level sources into a data stream;   processing the heterogeneous data streams to identify events therein, each event being identified by at least a unique ID, a timestamp, and an event type;   correlating the processed heterogeneous data streams to form an integrated data stream comprising a plurality of identified events;   detecting the existence and at least one characteristic of an anomaly in the computer network by application of a predetermined model of normalcy and one or more anomaly rules to the integrated data stream comprising the plurality of identified events; and   issuing an alert based on the at least one characteristic of the anomaly.   
     
     
         20 . The method of  claim 19  wherein the second level sources include one or more selected from the group consisting of data access, user behavior, computer activity and network activity.