Method and apparatus for recursively analyzing log file data in a network

Abstract

Method and apparatus for processing log data produced by a network is described. In one example, entries in the log data are filtered using a plurality of filters to select first entries from the entries. The first entries are filtered using a plurality of false positive filters associated with the plurality of filters to select second entries from the first entries. Unique IP addresses are identified in the second entries. The entries in the log data are then filtered using the unique IP addresses to select third set entries. The third entries are analyzed to detect one or more patterns.

1 . A method of processing log data produced by a network, comprising:
 filtering entries in the log data using a plurality of filters to select first entries from the entries;   filtering the first entries using a plurality of false positive filters associated with the plurality of filters to select second entries from the first entries;   identifying internet protocol (IP) addresses in the second entries;   filtering the entries in the log data using the IP addresses to select third entries from the entries; and   analyzing the third entries to detect one or more patterns.   
   
   
       2 . The method of  claim 1 , wherein the identifying comprises:
 separating the second entries into a plurality of streams based on a respective plurality of event types;   for each stream of the plurality of streams:
 identifying a set of IP addresses in the each stream of the plurality of streams; and 
 obtaining corresponding machine names for each IP address in the set of IP addresses. 
   
   
   
       3 . The method of  claim 2 , further comprising:
 formatting the each stream of the plurality of streams into a delimited format prior to the identifying of the set of IP addresses in the each stream of the plurality of streams.   
   
   
       4 . The method of  claim 3 , further comprising at least one of:
 storing results of the formatting for the stream in a first results file; and   storing results of the obtaining for the stream in a second results file.   
   
   
       5 . The method of  claim 1 , wherein the analyzing comprises:
 determining a count of the third entries corresponding to each of a plurality of event types; and   storing results of the determining in a results file.   
   
   
       6 . The method of  claim 1 , further comprising:
 storing results of the filtering using the sets of IP addresses in a results file.   
   
   
       7 . The method of  claim 1 , further comprising:
 storing results of the analyzing in a results file;   time stamping the results file; and   digitally signing the results file using a cryptographic key.   
   
   
       8 . The method of  claim 1 , wherein the log data comprises a plurality of logs produced by one or more internet protocol (IP) devices in the network. 
   
   
       9 . Apparatus for processing log data produced by a network, comprising:
 means for filtering entries in the log data using a plurality of filters to select first entries from the entries;   means for filtering the first entries using a plurality of false positive filters associated with the plurality of filters to select second entries from the first entries;   means for identifying internet protocol (IP) addresses in the second entries;   means for filtering the entries in the log data using the IP addresses to select third entries from the entries; and   means for analyzing the third entries to detect one or more patterns.   
   
   
       10 . The apparatus of  claim 9 , wherein the means for identifying comprises:
 means for separating the second entries into a plurality of streams based on a respective plurality of event types;   means for identifying a set of IP addresses for each stream of the plurality of streams; and   means for obtaining corresponding machine names for each IP address in the set of IP addresses for each stream of the plurality of streams.   
   
   
       11 . The apparatus of  claim 10 , further comprising:
 means for formatting the each stream of the plurality of streams into a delimited format prior to the identifying the set of IP addresses in the each stream of the plurality of streams.   
   
   
       12 . The apparatus of  claim 11 , further comprising at least one of:
 means for storing results of the formatting for the stream in a first results file; and   means for storing results of the obtaining for the stream in a second results file.   
   
   
       13 . The apparatus of  claim 9 , wherein the means for analyzing comprises:
 means for determining a count of the third entries corresponding to each of a plurality of event types; and   means for storing results of the determining in a results file.   
   
   
       14 . The apparatus of  claim 9 , further comprising:
 means for storing results of the filtering using the sets of IP addresses in a results file.   
   
   
       15 . A computer readable medium having stored thereon instructions that, when executed by a processor, cause the processor to perform a method of processing log data produced by a network, comprising:
 filtering entries in the log data using a plurality of filters to select first entries from the entries;   filtering the first entries using a plurality of false positive filters associated with the plurality of filters to select second entries from the first entries;   identifying internet protocol (IP) addresses in the second entries;   filtering the entries in the log data using the IP addresses to select third entries from the entries; and   analyzing the third entries to detect one or more patterns.   
   
   
       16 . The computer readable medium of  claim 15 , wherein the identifying comprises:
 separating the second entries into a plurality of streams based on a respective plurality of event types;   for each stream of the plurality of streams:
 identifying a set of IP addresses in the each stream of the plurality of streams; and 
 obtaining corresponding machine names for each IP address in the set of IP addresses. 
   
   
   
       17 . The computer readable medium of  claim 16 , further comprising:
 formatting the each stream of the plurality of streams into a delimited format prior to the identifying the set of IP addresses in the each stream of the plurality of streams.   
   
   
       18 . The computer readable medium of  claim 17 , further comprising at least one of:
 storing results of the formatting or the stream in a first results file; and   storing results of the obtaining for the stream in a second results file.   
   
   
       19 . The computer readable medium of  claim 15 , wherein the analyzing comprises:
 determining a count of the third entries corresponding to each of a plurality of event types; and   storing results of the determining in a results file.   
   
   
       20 . The computer readable medium of  claim 15 , further comprising:
 storing results of the analyzing in a results file;   time stamping the results file; and   digitally signing the results file using a cryptographic key.