Systems and methods for capturing, replaying, or analyzing time-series data

Abstract

Provided is an intrusion detection system configured to detect anomalies indicative of a zero-day attack by statistically analyzing substantially all traffic on a network in real-time. The intrusion detection system, in some aspects, includes a network interface; one or more processors communicatively coupled to the network interface; system memory communicatively coupled to the processors. The system memory, in some aspects, stores instructions that when executed by the processors cause the processors to perform steps including: buffering network data from the network interface in the system memory; retrieving the network data buffered in the system memory; applying each of a plurality of statistical or machine-learning intrusion-detection models to the retrieved network data; aggregating intrusion-likelihood scores from each of the intrusion-detection models in an aggregate score, and upon the aggregate score exceeding a threshold, outputting an alert.

1 . An intrusion detection system configured to detect anomalies indicative of a zero-day attack by statistically analyzing substantially all traffic on a network in real-time, the intrusion detection system comprising:
 a network interface;   one or more processors communicatively coupled to the network interface;   system memory communicatively coupled to the processors and storing instructions that when executed by the processors cause the processors to perform steps comprising:
 buffering network data from the network interface in the system memory; 
 retrieving the network data buffered in the system memory; 
 applying each of a plurality of statistical or machine-learning intrusion-detection models to the retrieved network data; 
 aggregating intrusion-likelihood scores from each of the intrusion-detection models in an aggregate score, and 
 upon the aggregate score exceeding a threshold, outputting an alert. 
   
     
     
         2 . The intrusion detection system of  claim 1 , wherein the plurality of statistical or machine-learning intrusion-detection models comprise distributed denial of service detection models and deep packet inspection models. 
     
     
         3 . The intrusion detection system of  claim 2 , wherein the distributed denial of service detection models comprise a packet-size distribution model, and a packet aggregate count distribution model. 
     
     
         4 . The intrusion detection system of  claim 2 , wherein the deep packet inspection models comprise an n-gram analysis model and a self-organizing map. 
     
     
         5 . The intrusion detection system of  claim 1 , wherein the network interface includes a one-gigabit per second Ethernet network interface. 
     
     
         6 . The intrusion detection system of  claim 1 , wherein the network interface includes a ten-gigabit per second Ethernet network interface. 
     
     
         7 . The intrusion detection system of  claim 1 , wherein buffering network data is performed concurrent with applying each of a plurality of statistical or machine-learning intrusion-detection models. 
     
     
         8 . (canceled) 
     
     
         9 . (canceled) 
     
     
         10 . (canceled) 
     
     
         11 . (canceled) 
     
     
         12 . (canceled) 
     
     
         13 . (canceled) 
     
     
         14 . An intrusion detection system comprising:
 a network interface;   one or more processors communicatively coupled to the network interface;   system memory communicatively coupled to the processors; and   system storage communicatively coupled to the processors, wherein the system storage stores instructions that when executed by the processors cause the intrusion detection system to perform steps comprising:
 writing network data from the network interface to a buffer in the system memory; and 
 concurrent with writing the network data to the buffer in the system memory, writing the network data from the buffer in the system memory to the system storage. 
   
     
     
         15 . The intrusion detection system of  claim 14 , comprising:
 prior to writing the network data to the system storage, ascertaining that more than a threshold amount of the network data is stored in the buffer in the system memory and has not yet been written to the system storage.   
     
     
         16 . The intrusion detection system of  claim 14 , comprising:
 prior to writing the network data to the system storage, ascertaining that more than a threshold duration of time has elapsed since the network data being written to the system storage was stored in the buffer in system memory.   
     
     
         17 . The intrusion detection system of  claim 14 , wherein the buffer in the system memory comprises a plurality of sub-buffers. 
     
     
         18 . The intrusion detection system of  claim 14 , wherein the plurality of sub-buffers are a circular sequence of buffers through which a write pointer cycles. 
     
     
         19 . The intrusion detection system of  claim 14 , wherein writing the network data from the network interface to the buffer in the system memory comprises:
 writing the network data to an active unlocked sub-buffer among the plurality of sub-buffers,   locking the active sub-buffer,   designating an unlocked sub-buffer as the active sub-buffer, and   after ascertaining that the network data stored in the locked sub-buffer has been written to system storage, unlocking the locked sub-buffer.   
     
     
         20 . (canceled) 
     
     
         21 . (canceled) 
     
     
         22 . (canceled) 
     
     
         23 . (canceled) 
     
     
         24 . (canceled) 
     
     
         25 . (canceled) 
     
     
         26 . (canceled) 
     
     
         27 . (canceled) 
     
     
         28 . (canceled) 
     
     
         29 . An intrusion detection system, comprising:
 a network interface;   a plurality of processors communicatively coupled to the network interface;   system memory communicatively coupled to the plurality of processors;   system storage communicatively coupled to the plurality of processors, wherein the system storage stores previously captured network data and instructions that when executed by the plurality of processors cause the intrusion detection system to perform steps comprising:
 pre-processing the network data in the system storage by calculating offsets defining batches of the network data sized such that each batch substantially fills a buffer; 
 moving a batch of the network data from the system storage to a buffer in the system memory, wherein the batch is stored between a pair of the calculated offsets in the system storage prior to moving; 
 concurrent with writing network data to the buffer, sending network data in the buffer to the network interface, wherein sending the network data comprises:
 reading time-stamps associated with packets in the network data, and 
 sending each packet when the associated time stamp corresponds with a time stamp counter of at least one of the processors. 
 
   
     
     
         30 . The intrusion detection system of  claim 29 , wherein the offsets are calculated such that packets in the network data do not overflow the buffer. 
     
     
         31 . The intrusion detection system of  claim 29 , wherein the buffer in the system memory comprises a plurality of sub-buffers. 
     
     
         32 . The intrusion detection system of  claim 31 , wherein the plurality of sub-buffers are a circular sequence of buffers through which a write pointer cycles. 
     
     
         33 . The intrusion detection system of  claim 31 , wherein the offsets are calculated such that packets in the network data do not overflow a sub-buffer among the plurality of sub-buffers. 
     
     
         34 . The intrusion detection system of  claim 31 , wherein sending the network data comprises:
 sending the network data from an active sub-buffer among the plurality of sub-buffers to the network interface,   designating the active sub-buffer as an exhausted sub-buffer,   designating another sub-buffer as the active sub-buffer, and   after ascertaining that new network data is stored in the exhausted sub-buffer, designating the exhausted sub-buffer as a replenished buffer.   
     
     
         35 . The intrusion detection system of  claim 34 , wherein moving the batch of the network data from the system storage to the buffer in the system memory comprises moving the batch of the network data to the exhausted sub-buffer. 
     
     
         36 . (canceled) 
     
     
         37 . (canceled) 
     
     
         38 . (canceled) 
     
     
         39 . (canceled) 
     
     
         40 . (canceled) 
     
     
         41 . (canceled) 
     
     
         42 . (canceled)