Path scanning for the detection of anomalous subgraphs and use of dns requests and host agents for anomaly/change detection and network situational awareness

Abstract

A system, apparatus, computer-readable medium, and computer-implemented method are provided for detecting anomalous behavior in a network. Historical parameters of the network are determined in order to determine normal activity levels. A plurality of paths in the network are enumerated as part of a graph representing the network, where each computing system in the network may be a node in the graph and the sequence of connections between two computing systems may be a directed edge in the graph. A statistical model is applied to the plurality of paths in the graph on a sliding window basis to detect anomalous behavior. Data collected by a Unified Host Collection Agent (“UHCA”) may also be used to detect anomalous behavior.

1 . A computer-implemented method, comprising:
 applying, by the computing system, an edge resolution model to a plurality of enumerated k-paths of sequences of directed edges in a graph representing connections in a computer network on a sliding window basis; and   detecting, by the computing system, anomalous behavior based on the applied edge resolution model.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the edge resolution model comprises an Observed Markov Model (“OMM”) or a Hidden Markov Model (“HMM”). 
     
     
         3 . The computer-implemented method of  claim 1 , wherein the plurality of enumerated k-paths comprise 3-paths. 
     
     
         4 . The computer-implemented method of  claim 1 , further comprising:
 determining, by the computing system, historical parameters of the network to determine normal activity levels, wherein   the computing system determines the historical parameters by taking into account at least two edge types.   
     
     
         5 . The computer-implemented method of  claim 4 , wherein a first edge type of the at least two edge types comprises member edges having sufficient data to estimate an individual model, and a second edge type of the at least two edge types comprises member edges where there is not sufficient data to estimate individual models for the member edges. 
     
     
         6 . The computer-implemented method of  claim 5 , wherein the second edge type is parameterized by a mean vector to ensure that models are not overly sensitive to low count edges. 
     
     
         7 . The computer-implemented method of  claim 1 , further comprising:
 collecting data, by the computing system, from a plurality of host agents pertaining to network communications sent and received by respective hosts in the network; and   analyzing the collected data, by the computing system, to detect anomalous behavior during a predetermined time period.   
     
     
         8 . An apparatus, comprising:
 at least one processor; and   memory storing computer program instructions, wherein the instructions, when executed by the at least one processor, are configured to cause the at least one processor to:
 apply a statistical model to a plurality of enumerated k-paths of sequences of directed edges in a graph representing connections in a computer network on a sliding window basis, and 
 detect anomalous behavior based on the applied statistical model. 
   
     
     
         9 . The apparatus of  claim 8 , wherein the statistical model comprises an Observed Markov Model (“OMM”) or a Hidden Markov Model (“HMM”). 
     
     
         10 . The apparatus of  claim 8 , wherein the plurality of enumerated k-paths comprise 3-paths. 
     
     
         11 . The apparatus of  claim 8 , wherein the computer program instructions are further configured to cause the at least one processor to determine historical parameters of the network to determine normal activity levels by taking into account at least two edge types. 
     
     
         12 . The apparatus of  claim 11 , wherein a first edge type of the at least two edge types comprises member edges having sufficient data to estimate an individual model, and a second edge type of the at least two edge types comprises member edges where there is not sufficient data to estimate individual models for the member edges. 
     
     
         13 . The apparatus of  claim 12 , wherein the second edge type is parameterized by a mean vector to ensure that models are not overly sensitive to low count edges. 
     
     
         14 . The apparatus of  claim 8 , wherein the computer program instructions are further configured to cause the at least one processor to:
 collect data from a plurality of host agents pertaining to network communications sent and received by respective hosts in the network, and   analyze the collected data to detect anomalous behavior during a predetermined time period.   
     
     
         15 . A system, comprising:
 memory storing computer program instructions; and   a plurality of processing cores configured to execute the stored computer program instructions, wherein the plurality of processing cores is configured to:
 apply a statistical model to a plurality of enumerated k-paths of sequences of directed edges in a graph representing connections in a computer network on a sliding window basis, and 
 detect anomalous behavior based on the applied statistical model. 
   
     
     
         16 . The system of  claim 15 , wherein the statistical model comprises an Observed Markov Model (“OMM”) or a Hidden Markov Model (“HMM”). 
     
     
         17 . The system of  claim 15 , wherein the plurality of enumerated k-paths comprise 3-paths. 
     
     
         18 . The system of  claim 15 , wherein
 the plurality of processing cores is further configured to determine historical parameters by taking into account at least two edge types,   a first edge type of the at least two edge types comprises member edges having sufficient data to estimate an individual model, and   a second edge type of the at least two edge types comprises member edges where there is not sufficient data to estimate individual models for the member edges.   
     
     
         19 . The system of  claim 18 , wherein the second edge type is parameterized by a mean vector to ensure that models are not overly sensitive to low count edges. 
     
     
         20 . The system of  claim 15 , wherein the plurality of processing cores is further configured to:
 collect data from a plurality of host agents pertaining to network communications sent and received by respective hosts in the network, and   analyze the collected data to detect anomalous behavior during a predetermined time period.   
     
     
         21 . A computer-implemented method, comprising:
 analyzing, by the computing system, collected data pertaining to network communications for each host of a plurality of hosts in a network to detect anomalous behavior during a predetermined time period by applying a statistical model to a plurality of enumerated k-paths of sequences of directed edges in a graph representing connections in a computer network on a sliding window basis; and   when anomalous behavior is detected, providing, by the computing system, an indication that the anomalous behavior occurred during the predetermined time period.   
     
     
         22 . The computer-implemented method of  claim 21 , wherein the collected data is sent as one-way communications from the host agents via User Datagram Protocol (“UDP”). 
     
     
         23 . The computer-implemented method of  claim 21 , wherein the data collected for each host comprises process stop and start information with checksums of starting process images, network connection event logs, a mapping of running processes to established network connections, and a current network connection state. 
     
     
         24 . The computer-implemented method of  claim 21 , wherein the collected data comprises a list of triples of values indicating network communication between hosts, each triple comprising a time when the communication occurred, a source Internet Protocol (“IP”) address, and a destination IP address. 
     
     
         25 . The computer-implemented method of  claim 21 , wherein the collecting of the data further comprises periodically polling the host agents for the data. 
     
     
         26 . The computer-implemented method of  claim 21 , further comprising:
 using, by the computing system, a Transmission Control Protocol (“TCP”) time wait state to collect information on short duration connections.   
     
     
         27 . The computer-implemented method of  claim 21 , further comprising:
 establishing, by the computing system, count weights using count information by calculating mean and variance statistics on counts.   
     
     
         28 . The computer-implemented method of  claim 21 , wherein the data is collected proportionally to a level of anomalousness on a respective host,
 at a low level of anomalousness, as deemed by deviation from a baseline probabilistic approach, the computing system collects basic network connectivity and process information.   at a moderate level of anomalousness, the computing system collects more process accounting and services and more complete network behavioral data, and   at a high level of anomalousness, the computing system collects full host behavioral information and performs full packet capture.