Systems and methods for capturing or analyzing time-series data

Abstract

Provided is an intrusion detection system configured to detect anomalies indicative of a zero-day attack by statistically analyzing substantially all traffic on a network in real-time. The intrusion detection system, in some aspects, includes a network interface; one or more processors communicatively coupled to the network interface; system memory communicatively coupled to the processors. The system memory, in some aspects, stores instructions that when executed by the processors cause the processors to perform steps including: buffering network data from the network interface in the system memory; retrieving the network data buffered in the system memory; applying each of a plurality of statistical or machine-learning intrusion-detection models to the retrieved network data; aggregating intrusion-likelihood scores from each of the intrusion-detection models in an aggregate score, and upon the aggregate score exceeding a threshold, outputting an alert.

1 . An intrusion detection system configured to detect anomalies indicative of a zero-day attack by statistically analyzing substantially all traffic on a network in real-time, the intrusion detection system comprising:
 a network interface;   one or more processors communicatively coupled to the network interface;   system memory communicatively coupled to the processors and storing instructions that when executed by the processors cause the processors to perform steps comprising:
 buffering network data from the network interface in the system memory; 
 retrieving the network data buffered in the system memory; 
 applying each of a plurality of statistical or machine-learning intrusion-detection models to the retrieved network data; 
 aggregating intrusion-likelihood scores from each of the intrusion-detection models in an aggregate score, and 
 upon the aggregate score exceeding a threshold, outputting an alert, 
   wherein the intrusion detection system is a single computer, and wherein the one or more processors comprise:   a central processing unit (CPU) upon which some of the intrusion detection models execute; and   a graphics processing unit (GPU) upon which some of the intrusion detection models execute, wherein both the CPU and the GPU communicate with the same system memory via memory channels on a motherboard to which the CPU and the GPU are physically attached.   
     
     
         2 . The intrusion detection system of  claim 1 , wherein the plurality of statistical or machine-learning intrusion-detection models comprise distributed denial of service detection models and deep packet inspection models. 
     
     
         3 . The intrusion detection system of  claim 2 , wherein the distributed denial of service detection models comprise a packet-size distribution model, and a packet aggregate count distribution model. 
     
     
         4 . The intrusion detection system of  claim 2 , wherein the deep packet inspection models comprise an n-gram analysis model and a self-organizing map. 
     
     
         5 . The intrusion detection system of  claim 1 , wherein the network interface includes a one-gigabit per second Ethernet network interface. 
     
     
         6 . The intrusion detection system of  claim 1 , wherein the network interface includes a ten-gigabit per second Ethernet network interface. 
     
     
         7 . The intrusion detection system of  claim 1 , wherein buffering network data is performed concurrent with applying each of a plurality of statistical or machine-learning intrusion-detection models. 
     
     
         8 . The intrusion detection system of  claim 1 , wherein retrieving the network data buffered in the system memory comprises:
 transferring the buffered network data from at least one sub-buffer among a plurality of sub-buffers to graphics memory of one or more graphics processing units,   wherein transferring is performed by a transferring thread or process and buffering is performed by buffering thread or process that is different from the transferring thread or process.   
     
     
         9 . The intrusion detection system of  claim 8 , wherein applying each of a plurality of statistical or machine-learning intrusion-detection models comprises:
 applying an n-gram model to the network data transferred to the one or more graphics processing units.   
     
     
         10 . The intrusion detection system of  claim 8 , wherein applying each of a plurality of statistical or machine-learning intrusion-detection models comprises:
 applying a self-organizing map to the network data transferred to the one or more graphics processing units.   
     
     
         11 . The intrusion detection system of  claim 1 , wherein buffering network data from the network interface in the system memory comprises associating a packet of the network data with a timestamp indicative of when the packet was received. 
     
     
         12 . A tangible non-transitory machine-readable medium storing instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:
 buffering network data from a network interface in system memory;   retrieving the network data buffered in the system memory;   applying each of a plurality of statistical or machine-learning intrusion-detection models to the retrieved network data;   aggregating intrusion-likelihood scores from each of the intrusion-detection models in an aggregate score, and   upon the aggregate score exceeding a threshold, outputting an alert,   wherein each of a plurality of statistical or machine-learning intrusion-detection models are applied within a single computing device using multiple types of processors of the single computing device, the multiple types of processors including a central processing unit that applies some of the statistical or machine-learning intrusion-detection models and aggregates intrusion-likelihood scores from each of the intrusion-detection models in an aggregate score.   
     
     
         13 . A method, comprising:
 buffering network data from a network interface in a system memory of a computing device, the system memory being dynamic random access memory physically attached to a system board of the computing device;   retrieving, with a central processing unit of the computing device that is physically attached to the system board, the network data buffered in the system memory;   applying each of a plurality of statistical or machine-learning intrusion-detection models to the retrieved network data, wherein some of the statistical or machine-learning intrusion-detection models are applied at least partially with the central processing unit and some of the statistical or machine-learning intrusion-detection models are applied at least partially with a graphics processing unit;   aggregating intrusion-likelihood scores from each of the intrusion-detection models in an aggregate score, and   upon the aggregate score exceeding a threshold, outputting an alert.   
     
     
         14 . An intrusion detection system comprising:
 a network interface;   one or more processors communicatively coupled to the network interface;   system memory communicatively coupled to the processors, wherein system memory includes dynamic random access memory; and   system storage communicatively coupled to the processors, wherein the system storage includes persistent storage operative to store data when power is removed, and wherein the system storage or system memory stores instructions that when executed by the processors cause the intrusion detection system to perform steps comprising:
 writing network data from the network interface to a buffer in the system memory; and 
 concurrent with writing the network data to the buffer in the system memory, writing the network data from the buffer in the system memory to the system storage, 
   wherein writing the network data from the network interface to the buffer in the system memory comprises:
 writing the network data to an active unlocked sub-buffer among the plurality of sub-buffers, 
 locking the active sub-buffer, 
 designating an unlocked sub-buffer as the active sub-buffer, and 
 determining that the network data stored in the locked sub-buffer has been written to system storage and, in response, unlocking the locked sub-buffer. 
   
     
     
         15 . The intrusion detection system of  claim 14 , comprising:
 prior to writing the network data to the system storage, ascertaining that more than a threshold amount of the network data is stored in the buffer in the system memory and has not yet been written to the system storage.   
     
     
         16 . The intrusion detection system of  claim 14 , comprising:
 prior to writing the network data to the system storage, ascertaining that more than a threshold duration of time has elapsed since the network data being written to the system storage was stored in the buffer in system memory.   
     
     
         17 . The intrusion detection system of  claim 14 , wherein the buffer in the system memory comprises a plurality of sub-buffers. 
     
     
         18 . The intrusion detection system of  claim 14 , wherein the plurality of sub-buffers are a circular sequence of buffers through which a write pointer cycles. 
     
     
         19 . (canceled) 
     
     
         20 . The intrusion detection system of  claim 14 , wherein locking the active sub-buffer comprises locking the active sub-buffer with a mutex. 
     
     
         21 . The intrusion detection system of  claim 14 , locking the active sub-buffer comprises locking the active sub-buffer with a spinlock. 
     
     
         22 . The intrusion detection system of  claim 14 , wherein locking the active sub-buffer comprises changing a variable state. 
     
     
         23 . The intrusion detection system of  claim 14 , wherein locking the active sub-buffer comprises incrementing or decrementing a semaphore. 
     
     
         24 . The intrusion detection system of  claim 14 , wherein writing the network data from the buffer in the system memory to the system storage, comprises:
 identifying a pre-existing file in the system storage; and   writing the network data from the buffer in the system memory to the pre-existing file in the system storage.   
     
     
         25 . The intrusion detection system of  claim 14 , wherein writing the network data from the buffer in the system memory to the system storage, comprises concurrently writing different portions of the network data from the buffer in the system memory to each of a plurality of storage drives. 
     
     
         26 . The intrusion detection system of  claim 25 , wherein concurrently writing the network data from the buffer in the system memory to each of a plurality of storage drives comprises writing to eight or more hard disk drives. 
     
     
         27 . The intrusion detection system of  claim 1 , wherein:
 the single computer is a rack-mountable computing device,   buffering network data from the network interface in the system memory comprises transferring data into random access memory via direct memory access from the network interface,   the system memory is dynamic random access memory,   wherein the CPU, the GPU, and the mother board are commodity computing hardware, and   wherein the instructions cause the network data to be written from the system memory into graphics memory of the GPU.   
     
     
         28 . The intrusion detection system of  claim 1 , wherein:
 buffering network data from the network interface in the system memory comprises:
 writing the network data from the network interface to a sub-buffer identified by a write pointer, the sub-buffer being one sub-buffer among a plurality of sub-buffers; and 
   retrieving the network data buffered in the system memory comprises:
 determining whether a read pointer identifies a sub-buffer that is locked; and 
 in response to determining that the read pointer identifies a sub-buffer that is locked, writing the network data in the identified sub-buffer to graphics memory of the GPU. 
   
     
     
         29 . The intrusion detection system of  claim 28 , comprising:
 determining that the sub-buffer identified by the write pointer is full in response to an occupied space of the sub-buffer identified by the write pointer exceeding a threshold based on a buffer size less a maximum specified packet size to prevent frames from extending beyond boundaries of the sub-buffers.   
     
     
         30 . The system of  claim 14 , wherein locking the active sub-buffer comprises:
 determining that the active sub-buffer is full in response to an occupied space of the active sub-buffer exceeding a threshold based on a buffer size less a maximum specified packet size to prevent frames from extending beyond boundaries of the sub-buffers.