Systems and methods for capturing or replaying time-series data

Abstract

Provided is an intrusion detection system configured to detect anomalies indicative of a zero-day attack by statistically analyzing substantially all traffic on a network in real-time. The intrusion detection system, in some aspects, includes a network interface; one or more processors communicatively coupled to the network interface; system memory communicatively coupled to the processors. The system memory, in some aspects, stores instructions that when executed by the processors cause the processors to perform steps including: buffering network data from the network interface in the system memory; retrieving the network data buffered in the system memory; applying each of a plurality of statistical or machine-learning intrusion-detection models to the retrieved network data; aggregating intrusion-likelihood scores from each of the intrusion-detection models in an aggregate score, and upon the aggregate score exceeding a threshold, outputting an alert.

1 . A tangible non-transitory machine-readable medium storing instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:
 writing network data from a network interface to a buffer in the system memory, wherein writing the network data from the network interface to the buffer in the system memory comprises:
 writing the network data to an active unlocked sub-buffer among the plurality of sub-buffers, 
 locking the active sub-buffer, 
 designating an unlocked sub-buffer as the active sub-buffer, and 
 after ascertaining that the network data stored in the locked sub-buffer has been written to system storage, unlocking the locked sub-buffer; and 
   concurrent with writing the network data to the buffer in the system memory, writing the network data from the buffer in the system memory to the system storage.   
     
     
         2 . A method, comprising:
 writing network data from a network interface to a buffer in the system memory, wherein writing the network data from the network interface to the buffer in the system memory comprises:
 writing the network data to an active unlocked sub-buffer among the plurality of sub-buffers, 
 locking the active sub-buffer, 
 designating an unlocked sub-buffer as the active sub-buffer, and 
 after ascertaining that the network data stored in the locked sub-buffer has been written to system storage, unlocking the locked sub-buffer; and 
   concurrent with writing the network data to the buffer in the system memory, writing the network data from the buffer in the system memory to the system storage.   
     
     
         3 . (canceled) 
     
     
         4 . (canceled) 
     
     
         5 . (canceled) 
     
     
         6 . (canceled) 
     
     
         7 . (canceled) 
     
     
         8 . (canceled) 
     
     
         9 . (canceled) 
     
     
         10 . (canceled) 
     
     
         11 . (canceled) 
     
     
         12 . (canceled) 
     
     
         13 . (canceled) 
     
     
         14 . (canceled) 
     
     
         15 . (canceled) 
     
     
         16 . The method of  claim 2 , comprising:
 prior to writing the network data to the system storage, ascertaining that more than a threshold duration of time has elapsed since the network data being written to the system storage was stored in the buffer in system memory.   
     
     
         17 . The method of  claim 2 , wherein:
 the plurality of sub-buffers are a circular sequence of buffers through which a write pointer cycles, the write pointer identifying one of the plurality of sub-buffers.   
     
     
         18 . The method of  claim 2 , wherein locking the active sub-buffer comprises:
 locking the active sub-buffer with a mutex;   locking the active sub-buffer with a spinlock;   changing a variable state; or   incrementing or decrementing a semaphore.   
     
     
         19 . The method of  claim 2 , wherein writing the network data from the buffer in the system memory to the system storage, comprises:
 writing the network data from the buffer in the system memory to at least one preexisting file in the system storage.   
     
     
         20 . The method of  claim 2 , wherein writing the network data from the buffer in the system memory to the system storage, comprises:
 concurrently writing different portions of the network data from the buffer in the system memory to each of a plurality of storage drives.   
     
     
         21 . The method of  claim 20 , wherein concurrently writing the network data from the buffer in the system memory to each of a plurality of storage drives comprises:
 writing to eight or more hard disk drives.   
     
     
         22 . The method of  claim 2 , wherein writing network data from a network interface to a buffer in the system memory comprises:
 capturing all network traffic on a ten gigabit per second network with a single computing device.   
     
     
         23 . The method of  claim 2 , wherein writing the network data to an active unlocked sub-buffer among the plurality of sub-buffers comprises:
 associating packets with timestamps indicative of the time at which each packet was received.   
     
     
         24 . The method of  claim 2 , wherein writing the network data to an active unlocked sub-buffer among the plurality of sub-buffers comprises:
 ascertaining which sub-buffer among the plurality of sub-buffers in system memory is active according to a pointer to the active sub-buffer.   
     
     
         25 . The method of  claim 2 , wherein writing the network data to an active unlocked sub-buffer among the plurality of sub-buffers comprises:
 writing the network data to a portion of memory allocated to a network interface with a driver of the network interface with direct memory access to the portion of memory; and   writing the network data to the active unlocked sub-buffer from the portion of memory.   
     
     
         26 . The method of  claim 2 , wherein locking the active sub-buffer comprises:
 determining that the active sub-buffer is full in response to an occupied space of the active sub-buffer exceeding a size of the active buffer less a maximum specified packet size.   
     
     
         27 . The method of  claim 2 , wherein designating an unlocked sub-buffer as the active sub-buffer comprises:
 changing a value of a write pointer that identifies one of the plurality of sub-buffers.   
     
     
         28 . The method of  claim 2 , wherein writing the network data from the buffer in the system memory to the system storage comprises:
 identifying a locked sub-buffer with a read pointer; and   writing network data in the locked sub-buffer to system storage.   
     
     
         29 . The method of  claim 2 , wherein writing network data in the locked sub-buffer to system storage comprises writing multiple frames of network data in the locked sub-buffer to system storage with one and only one write command. 
     
     
         30 . The method of  claim 2 , wherein writing the network data from the buffer in the system memory to the system storage comprises:
 identifying a pre-formed file in system storage and overwriting the file.   
     
     
         31 . The method of  claim 2 , comprising:
 replaying the network data stored in the system storage; and   testing an intrusion detection system with the replayed network data.   
     
     
         32 . The method of  claim 30 , comprising:
 adjusting parameters of an intrusion detection system in response to the testing.