Context-based identification of anomalous log data

Abstract

Disclosed herein are methods, systems, and processes for context-based identification of anomalous log data. Log data with multiple original logs is received at an anomalous log data identification system. A context associated training dataset is generated by splitting a string in a log into multiple split strings, generating a context association between each split string and a unique key that corresponds to the log, and generating an input/output (I/O) string data batch that includes I/O string data for each split string in the log by training each split string against every other split string in the log. A context-based anomalous log data identification model is then trained according to a machine learning technique using the I/O string data batch that includes a list of unique strings in the context associated training dataset. The training tunes the context-based anomalous log data identification model to classify or cluster a vector associated with a new string in a new log that is not part of the multiple original logs as anomalous.

What is claimed is: 
     
       1. A computer-implemented method, comprising:
 performing, by one or more hardware processors with associated memory that implement a context-based anomalous log data identification system: 
 receiving log data comprising a plurality of logs; 
 generating a context associated training dataset, comprising
 splitting a string in a log of the plurality of logs into a plurality of split strings, 
 generating a context association between each of the plurality of split strings and a unique key that corresponds to the log, and 
 generating an input/output (I/O) string data batch comprising I/O string data for each split string in the log by training each split string against every other split string of the plurality of split strings in the log; and 
 
 training a context-based anomalous log data identification model using the I/O string data batch comprising a list of unique strings in the context associated training dataset and according to a machine learning technique, wherein
 the training tunes the context-based anomalous log data identification model to classify or cluster a vector associated with a new string in a new log that is not part of the plurality of logs as anomalous, 
 training the context-based anomalous log data identification model to perform cluster analysis is based on whether an executable that is part of the process information is a good executable that is part of a bad path, and 
 the good executable and the bad path are pre-identified based at least on a classifier prior to performing the cluster analysis. 
 
 
     
     
       2. The computer-implemented method of  claim 1 , further comprising:
 generating a dense vector for the log. 
 
     
     
       3. The computer-implemented method of  claim 2 , wherein
 generating the dense vector for the log comprises:
 accessing the list of unique split strings, and 
 averaging a plurality of vectors comprising at least one vector for each unique split string in the list of unique split strings, and 
 
 the dense vector indicates a mapping of each unique split string in the list of unique split strings to the dense vector being trained. 
 
     
     
       4. The computer-implemented method of  claim 3 , further comprising:
 training the context-based anomalous log data identification model with additional I/O string data generated by the context-based anomalous log data identification system for each log of the plurality of logs. 
 
     
     
       5. The computer-implemented method of  claim 1 , wherein
 the log data comprises process information associated with one or more computing systems generating the log data, and 
 the process information comprises a plurality of process names/hashes. 
 
     
     
       6. The computer-implemented method of  claim 5 , wherein
 training the context-based anomalous log data identification model to perform cluster analysis is based at least on a number of occurrences of a process name/hash of the plurality of process names/hashes in the log. 
 
     
     
       7. A non-transitory computer readable storage medium comprising program instructions executable to:
 perform, by one or more hardware processors with associated memory that implement a context-based anomalous log data identification system: 
 receive log data comprising a plurality of logs; 
 generate a context associated training dataset, comprising
 splitting a string in a log of the plurality of logs into a plurality of split strings, 
 generating a context association between each of the plurality of split strings and a unique key that corresponds to the log, and 
 generating an input/output (I/O) string data batch comprising I/O string data for each split string in the log by training each split string against every other split string of the plurality of split strings in the log; and 
 
 train a context-based anomalous log data identification model using the I/O string data batch comprising a list of unique strings in the context associated training dataset and according to a machine learning technique, wherein
 the training tunes the context-based anomalous log data identification model to classify or cluster a vector associated with a new string in a new log that is not part of the plurality of logs as anomalous, 
 training the context-based anomalous log data identification model to perform cluster analysis is based on whether an executable that is part of the process information is a good executable that is part of a bad path, and 
 the good executable and the bad path are pre-identified based at least on a classifier prior to performing the cluster analysis. 
 
 
     
     
       8. The non-transitory computer readable storage medium of  claim 7 , further comprising:
 generating a dense vector for the log. 
 
     
     
       9. The non-transitory computer readable storage medium of  claim 8 , wherein
 generating the dense vector for the log comprises:
 accessing the list of unique split strings, and 
 averaging a plurality of vectors comprising at least one vector for each unique split string in the list of unique split strings, and 
 
 the dense vector indicates a mapping of each unique split string in the list of unique split strings to the dense vector being trained. 
 
     
     
       10. The non-transitory computer readable storage medium of  claim 9 , further comprising:
 training the context-based anomalous log data identification model with additional I/O string data generated by the context-based anomalous log data identification system for each log of the plurality of logs. 
 
     
     
       11. The non-transitory computer readable storage medium of  claim 7 , wherein
 the log data comprises process information associated with one or more computing systems generating the log data, and 
 the process information comprises a plurality of process names/hashes. 
 
     
     
       12. The non-transitory computer readable storage medium of  claim 11 , wherein
 training the context-based anomalous log data identification model to perform cluster analysis is further based at least on a number of occurrences of a process name/hash of the plurality of process names/hashes in the log. 
 
     
     
       13. A system comprising:
 one or more processors; and 
 a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to: 
 perform, by one or more hardware processors with associated memory that implement a context-based anomalous log data identification system: 
 receive log data comprising a plurality of logs; 
 generate a context associated training dataset, comprising
 splitting a string in a log of the plurality of logs into a plurality of split strings, 
 generating a context association between each of the plurality of split strings and a unique key that corresponds to the log, and 
 generating an input/output (I/O) string data batch comprising I/O string data for each split string in the log by training each split string against every other split string of the plurality of split strings in the log; and 
 
 train a context-based anomalous log data identification model using the I/O string data batch comprising a list of unique strings in the context associated training dataset and according to a machine learning technique, wherein
 the training tunes the context-based anomalous log data identification model to classify or cluster a vector associated with a new string in a new log that is not part of the plurality of logs as anomalous, 
 training the context-based anomalous log data identification model to perform cluster analysis is based on whether an executable that is part of the process information is a good executable that is part of a bad path, and 
 the good executable and the bad path are pre-identified based at least on a classifier prior to performing the cluster analysis. 
 
 
     
     
       14. The system of  claim 13 , further comprising:
 generating a dense vector for the log, wherein 
 generating the dense vector for the log comprises:
 accessing the list of unique split strings, and 
 averaging a plurality of vectors comprising at least one vector for each unique split string in the list of unique split strings, and 
 
 the dense vector indicates a mapping of each unique split string in the list of unique split strings to the dense vector being trained. 
 
     
     
       15. The system of  claim 14 , further comprising:
 training the context-based anomalous log data identification model with additional I/O string data generated by the context-based anomalous log data identification system for each log of the plurality of logs. 
 
     
     
       16. The system of  claim 13 , wherein
 the log data comprises process information associated with one or more computing systems generating the log data, and 
 the process information comprises a plurality of process names/hashes. 
 
     
     
       17. The system of  claim 16 , wherein
 training the context-based anomalous log data identification model to perform cluster analysis is based at least on a number of occurrences of a process name/hash of the plurality of process names/hashes in the log.