Identifying behavioral patterns of events derived from machine data that reveal historical behavior of an information technology environment

Abstract

Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.

What is claimed is: 
     
       1. A method, comprising:
 receiving first machine data, the first machine data reflecting activity in an information technology environment; 
 generating signatures from portions of the first machine data; 
 receiving second machine data, the second machine data reflecting activity in the information technology environment; 
 identifying sources of the second machine data using the signatures generated from the portions of the first machine data, wherein the sources include a first computing device associated with a first signature of the signatures and a second computing device associated with a second signature of the signatures; 
 determining a first event from the second machine data based at least in part on the first signature, wherein the first event is associated with the first computing device; 
 determining a second event from the second machine data based at least in part on the second signature, wherein the second event is associated with the second computing device; 
 determining a behavioral pattern in real time between the first event and the second event based on a co-occurrence of the first event and the second event within a time window; 
 determining historical behavior of the information technology environment based at least in part on the determined behavioral pattern; and 
 sending at least a portion of the determined historical behavior to a computing system; 
 wherein the method is performed by one or more computing devices. 
 
     
     
       2. The method as recited in  claim 1 , wherein the behavioral pattern is a first behavioral pattern, the method further comprising determining an association between the first behavioral pattern and a second behavioral pattern, wherein the second behavioral pattern is between a third event from the second machine data and a fourth event from the second machine data, wherein the association between the first behavioral pattern and the second behavioral pattern corresponds to the determined historical behavior, wherein the determining the association between the first behavioral pattern and the second behavioral pattern comprises:
 identifying a timing pattern between the first behavioral pattern and the second behavioral pattern. 
 
     
     
       3. The method as recited in  claim 1 , wherein the behavioral pattern is a first behavioral pattern, the method further comprising determining an association between the first behavioral pattern and a second behavioral pattern between a third event from the second machine data and a fourth event from the second machine data, wherein the association between the first behavioral pattern and the second behavioral pattern corresponds to the determined historical behavior, wherein the determining the association between the first behavioral pattern and the second behavioral pattern comprises:
 identifying a timing pattern between the first behavioral pattern and the second behavioral pattern; and 
 measuring a statistical frequency of the timing pattern. 
 
     
     
       4. The method as recited in  claim 1 , wherein the behavioral pattern is a first behavioral pattern, the method further comprising determining an association between the first behavioral pattern and a second behavioral pattern between a third event from the second machine data and a fourth event from the second machine data, wherein the association between the first behavioral pattern and the second behavioral pattern corresponds to the determined historical behavior, wherein the determining the association between the first behavioral pattern and the second behavioral pattern comprises:
 identifying a timing pattern between the first behavioral pattern and the second behavioral pattern; 
 measuring a statistical frequency of the timing pattern; and 
 determining that the first behavioral pattern and the second behavioral pattern are associated based on the statistical frequency of the timing pattern. 
 
     
     
       5. The method as recited in  claim 1 , wherein a first portion of the second machine data that is produced from the first computing device has a different data format than a second portion of the second machine data produced from the second computing device. 
     
     
       6. The method as recited in  claim 1 , wherein determining the first event and determining the second event comprises:
 analyzing the second machine data in order to segment the second machine data into the first event associated with the first computing device and the second event associated with the second computing device by determining a beginning and an ending of each of the first event and the second event, each of the first event and the second event including a portion of the second machine data segmented for that event. 
 
     
     
       7. The method as recited in  claim 1 , further comprising:
 analyzing the second machine data in order to segment the second machine data into the first event associated with the first computing device and the second event associated with the second computing device by determining a beginning and an ending of each of the first event and the second event, each of the first event and the second event including a portion of the second machine data segmented for that event; and 
 associating a time stamp with each of the first event and the second event, the time stamp derived from second machine data segmented for that event. 
 
     
     
       8. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause the one or more processors to:
 receive first machine data, the first machine data reflecting activity in an information technology environment; 
 generate signatures from portions of the first machine data; 
 receive second machine data, the second machine data reflecting activity in the information technology environment; 
 identify sources of the second machine data using the signatures generated from the portions of the first machine data, wherein the sources include a first computing device associated with a first signature of the signatures and a second computing device associated with a second signature of the signatures; 
 determine a first event from the second machine data based at least in part on the first signature, wherein the first event is associated with the first computing device; 
 determine a second event from the second machine data based at least in part on the second signature, wherein the second event is associated with the second computing device; 
 determine a behavioral pattern in real time between the first event and the second event based on a co-occurrence of the first event and the second event within a time window; 
 determine historical behavior of the information technology environment based at least in part on the determined behavioral pattern; and 
 send at least a portion of the determined historical behavior to a computing system. 
 
     
     
       9. The one or more non-transitory computer-readable storage media as recited in  claim 8 , wherein the behavioral pattern is a first behavioral pattern, wherein the one or more sequences of instructions, when executed by the one or more processors further cause the one or more processors to determine an association between the first behavioral pattern and a second behavioral pattern between a third event from the second machine data and a fourth event from the second machine data, wherein the association between the first behavioral pattern and the second behavioral pattern corresponds to the determined historical behavior, wherein to determine the association between the first behavioral pattern and the second behavioral pattern, the one or more sequences of instructions, when executed by the one or more processors further cause the one or more processors to:
 identify a timing pattern between the first behavioral pattern and the second behavioral pattern. 
 
     
     
       10. The one or more non-transitory computer-readable storage media as recited in  claim 8 , wherein the behavioral pattern is a first behavioral pattern, wherein the one or more sequences of instructions, when executed by the one or more processors further cause the one or more processors to determine an association between the first behavioral pattern and a second behavioral pattern between a third event from the second machine data and a fourth event from the second machine data, wherein the association between the first behavioral pattern and the second behavioral pattern corresponds to the determined historical behavior, wherein to determine the association between the first behavioral pattern and the second behavioral pattern, the one or more sequences of instructions, when executed by the one or more processors further cause the one or more processors to:
 identify a timing pattern between the first behavioral pattern and the second behavioral pattern; and 
 measure a statistical frequency of the timing pattern. 
 
     
     
       11. The one or more non-transitory computer-readable storage media as recited in  claim 8 , wherein the behavioral pattern is a first behavioral pattern, wherein the one or more sequences of instructions, when executed by the one or more processors further cause the one or more processors to determine an association between the first behavioral pattern and a second behavioral pattern between a third event from the second machine data and a fourth event from the second machine data, wherein the association between the first behavioral pattern and the second behavioral pattern corresponds to the determined historical behavior, wherein to determine the association between the first behavioral pattern and the second behavioral pattern, the one or more sequences of instructions, when executed by the one or more processors further cause the one or more processors to:
 identify a timing pattern between the first behavioral pattern and the second behavioral pattern; 
 measure a statistical frequency of the timing pattern; and 
 determine that the first behavioral pattern and the second behavioral pattern are associated based on the statistical frequency of the timing pattern. 
 
     
     
       12. The one or more non-transitory computer-readable storage media as recited in  claim 8 , wherein a first portion of the second machine data produced from the first computing device has a different data format than a second portion of the second machine data produced from the second computing device. 
     
     
       13. The one or more non-transitory computer-readable storage media as recited in  claim 8 , wherein to determine the first event and to determine the second event, the one or more sequences of instructions, when executed by the one or more processors further cause the one or more processors to:
 analyze the second machine data in order to segment the second machine data into the first event associated with the first computing device and the second event associated with the second computing device by determining a beginning and an ending of each of the first event and the second event, each of the first event and the second event including a portion of the second machine data segmented for that event. 
 
     
     
       14. The one or more non-transitory computer-readable storage media as recited in  claim 8 , wherein to determine the first event and to determine the second event, the one or more sequences of instructions, when executed by the one or more processors further cause the one or more processors to:
 analyze the second machine data in order to segment the second machine data into the first event associated with the first computing device and the second event associated with the second computing device by determining a beginning and an ending of each of the first event and the second event, each of the first event and the second event including a portion of the second machine data segmented for that event; and 
 associate a time stamp with each of the first event and the second event, the time stamp derived from second machine data segmented for that event. 
 
     
     
       15. An apparatus, comprising:
 one or more processors; and 
 a memory storing instructions, which when executed by the one or more processors, causes the one or more processors to:
 receive first machine data, the first machine data reflecting activity in an information technology environment; 
 generate signatures from portions of the first machine data; 
 receive second machine data, the second machine data reflecting activity in the information technology environment; 
 identify sources of the second machine data using the signatures generated from the portions of the first machine data, wherein the sources include a first computing device associated with a first signature of the signatures and a second computing device associated with a second signature of the signatures; 
 determine a first event from the second machine data based at least in part on the first signature, wherein the first event is associated with the first computing device; 
 determine a second event from the second machine data based at least in part on the second signature, wherein the second event is associated with the second computing device; 
 determine a behavioral pattern in real time between the first event and the second event based on a co-occurrence of the first event and the second event within a time window; 
 determine historical behavior of the information technology environment based at least in part on the determined behavioral pattern; and 
 send at least a portion of the determined historical behavior to a computing system. 
 
 
     
     
       16. The apparatus as recited in  claim 15 , wherein the behavioral pattern is a first behavioral pattern, wherein the instructions, when executed by the one or more processors further cause the one or more processors to determine an association between the first behavioral pattern and a second behavioral pattern between a third event from the second machine data and a fourth event from the second machine data, wherein the association between the first behavioral pattern and the second behavioral pattern corresponds to the determined historical behavior, wherein to determine the association between the first behavioral pattern and the second behavioral pattern, the instructions, when executed by the one or more processors further cause the one or more processors to:
 identify a timing pattern between the behavioral pattern and the second behavioral pattern. 
 
     
     
       17. The apparatus as recited in  claim 15 , wherein the behavioral pattern is a first behavioral pattern, wherein the instructions, when executed by the one or more processors further cause the one or more processors to determine an association between the first behavioral pattern and a second behavioral pattern between a third event from the second machine data and a fourth event from the second machine data, wherein the association between the first behavioral pattern and the second behavioral pattern corresponds to the determined historical behavior, wherein to determine the association between the first behavioral pattern and the second behavioral pattern, the instructions, when executed by the one or more processors further cause the one or more processors to:
 identify a timing pattern between the first behavioral pattern and the second behavioral pattern; and 
 measure a statistical frequency of the timing pattern. 
 
     
     
       18. The apparatus as recited in  claim 15 , wherein the behavioral pattern is a first behavioral pattern, wherein the instructions, when executed by the one or more processors further cause the one or more processors to determine an association between the first behavioral pattern and a second behavioral pattern between a third event from the second machine data and a fourth event from the second machine data, wherein the association between the first behavioral pattern and the second behavioral pattern corresponds to the determined historical behavior, wherein to determine the association between the first behavioral pattern and the second behavioral pattern, the instructions, when executed by the one or more processors further cause the one or more processors to:
 identify a timing pattern between the first behavioral pattern and the second behavioral pattern; 
 measure a statistical frequency of the timing pattern; and 
 determine that the first behavioral pattern and the second behavioral pattern are associated based on the statistical frequency of the timing pattern. 
 
     
     
       19. The apparatus as recited in  claim 15 , wherein a first portion of the second machine data produced from the first computing device has a different data format than a second portion of the second machine data produced from the second computing device. 
     
     
       20. The apparatus as recited in  claim 15 , wherein to determine the first event and the second event, the instructions, when executed by the one or more processors further cause the one or more processors to:
 analyze the second machine data in order to segment the second machine data into the first event associated with the first computing device and the second event associated with the second computing device by determining a beginning and an ending of each of the first event and the second event, each of the first event and the second event including a portion of the second machine data segmented for that event; and 
 associate a time stamp with each of the first event and the second event, the time stamp derived from second machine data segmented for that event.