Systems and methods for machine learning assisted syslog parser generators

Abstract

Systems and methods for generating a parser from a log file including: receiving a log file, wherein the log file is a structured text file of a plurality of data elements; invoking a machine learning model to: process the log file to identify name-value-pairs from the data elements; classify the log file as being associated with a schema based in part on the name-value pairs; map a first name-value pair to a first input field of the schema based on characteristics of the first name-value pair; determine a confidence level associated with mapping the first name-value pair to the first input field; and when the confidence level for mapping the first name-value pair exceeds a threshold, provide the first name-value pair to the first input field; and generating a parser from the plurality of input fields of the schema.

1 . A computer-implemented method comprising:
 receiving a log file from a user device, wherein the log file is a structured text file of a plurality of data elements;   invoking one or more machine learning models configured to:
 process the log file to identify one or more name-value-pairs representing data associated with a cybersecurity threat event occurring at or detected by an affected device from the plurality of data elements; 
 classify the log file as being associated with a schema from a set of schemas based in part on the one or more name-value pairs; 
 map a first name-value pair of the one or more name-value pairs to a first input field from a plurality of input fields of the schema based on characteristics of the first name-value pair; 
 determine a confidence level associated with mapping the first name-value pair to the first input field; and 
 when the confidence level for mapping the first name-value pair to the first input field exceeds a threshold, provide the first name-value pair to the first input field; 
   generating a new parser from the plurality of input fields of the schema and the mapping using a parser algorithm associated with the schema, wherein the generated new parser includes at least part of the first name-value pair; and   using the new parser in a compiler to compile log files into a computer-readable format compatible with an application for identifying and evaluating cybersecurity threats from log files.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the method further comprises:
 receiving an edited parser, wherein edits to obtain the edited parser from an initial parser include one or more of: adding code to the initial parser, removing code from the initial parser, and adjusting values of the first name-value pair.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein the method further comprises:
 storing the edited parser in a repository of edited parsers associated with a user;   receiving a second log file; and   parsing the second log file using the edited parser.   
     
     
         4 . The computer-implemented method of  claim 1 , wherein the log file includes one or more name-value pairs associated with: a timestamp, a product, a product version, a vendor, a user, and a severity identifier indicating a cybersecurity threat. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein at least one name-value pair of the one or more name-value pairs includes terminology indicating a type of device used to collect telemetry data, network data, or a combination thereof associated with the cybersecurity threat event, wherein the type of device is used by the one or more machine learning models to classify the log file as being associated with the schema. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein the one or more machine learning models use one or more of: a naïve-bayes classifier and a random forest model to map the one or more name-value pairs to one or more input fields from the plurality of input fields of the schema. 
     
     
         7 . The computer-implemented method of  claim 2 , wherein the one or more machine learning models are further configured to be trained using the edited parser as intrinsic training data to improve classifying of log files and mapping of data elements to an associated schema. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein when the confidence level for mapping the first name-value pair to the first input field does not exceed the threshold, determine a confidence level for mapping a second name-value pair from the one or more name-value pairs to the first input field. 
     
     
         9 . The computer-implemented method of  claim 8 , wherein when no name-value pair of the one or more name-value pairs exceeds the threshold for mapping to the first input field, a portion of the new parser associated with the first input field of the schema is populated with a null value. 
     
     
         10 . A system comprising:
 a memory with instructions stored thereon; and   a processing device, coupled to the memory, the processing device configured to access the memory and execute the instructions, wherein the instructions cause the processing device to perform or control performance of operations comprising:
 receiving a log file from a user device, wherein the log file is a structured text file of a plurality of data elements; 
 invoking one or more machine learning models configured to:
 process the log file to identify one or more name-value-pairs representing data associated with a cybersecurity threat event occurring at or detected by an affected device from the plurality of data elements; 
 classify the log file as being associated with a schema from a set of schemas based in part on the one or more name-value pairs; 
 map a first name-value pair of the one or more name-value pairs to a first input field from a plurality of input fields of the schema from the set of schemas based on characteristics of the first name-value pair; 
 determine a confidence level associated with mapping the first name-value pair to the first input field; and 
 when the confidence level for mapping the first name-value pair to the first input field exceeds a threshold, provide the first name-value pair to the first input field; 
 
 generating a new parser from the plurality of input fields of the schema and the mapping using a parser algorithm associated with the schema, wherein the generated new parser includes at least part of the first name-value pair; and 
 using the new parser in a compiler to compile log files into a computer-readable format compatible with an application for identifying and evaluating cybersecurity threats from log files. 
   
     
     
         11 . The system of  claim 10 , wherein the operations further comprise:
 receiving an edited parser, wherein edits to obtain the edited parser from an initial parser include one or more of: adding code to the initial parser, removing code from the initial parser, and adjusting values of the first name-value pair.   
     
     
         12 . The system of  claim 11 , wherein the operations further comprise:
 storing the edited parser in a repository of edited parsers associated with a user;   receiving a second log file; and   parsing the second log file using the edited parser.   
     
     
         13 . The system of  claim 10 , wherein the log file includes one or more name-value pairs associated with: a timestamp, a product, a product version, a vendor, a user, and a severity identifier indicating a cybersecurity threat. 
     
     
         14 . The system of  claim 10 , wherein the one or more machine learning models use a nearest neighbor algorithm to classify the log file. 
     
     
         15 . The system of  claim 10 , wherein the one or more machine learning models use one or more of: a naïve-bayes classifier and a random forest model to map the one or more name-value pairs to one or more input fields from the plurality of input fields of the schema. 
     
     
         16 . The system of  claim 11 , wherein the one or more machine learning models are further configured to be trained using the edited parser as intrinsic training data to improve classifying of log files and mapping of data elements to an associated schema. 
     
     
         17 . The system of  claim 10 , wherein operations further comprise:
 when the confidence level for mapping the first name-value pair to the first input field does not exceed the threshold, determining a confidence level for mapping a second name-value pair from the one or more name-value pairs to the first input field.   
     
     
         18 . The system of  claim 17 , wherein when no name-value pair of the one or more name-value pairs exceeds the threshold for mapping to the first input field, a portion of the new parser associated with the first input field of the schema is populated with a null value. 
     
     
         19 . A system comprising:
 a memory with instructions stored thereon; and   a processing device, coupled to the memory, the processing device configured to access the memory and execute the instructions, wherein the instructions cause the processing device to perform or control performance of operations comprising:
 receiving, from a first user device, a log file including a plurality of name-value pairs representing data associated with a cybersecurity threat event occurring at or detected by an affected device; 
 tokenizing the plurality of name-value pairs of the log file; 
 generating a distribution of tokenized name-value pairs; 
 classifying the log file based on the distribution of tokenized name-value pairs; 
 generating a feature vector associated with one or more of the tokenized name-value pairs, wherein attributes of the feature vector include the one or more of the tokenized name-value pairs; 
 providing the feature vector to a plurality of decision trees to determine a confidence level associated with mapping name-value pairs of the plurality of name-value pairs to input fields of a schema, wherein when the confidence level exceeds a threshold, the operations further comprise mapping one or more name-value pairs of the plurality of name-value pairs to associated input fields of the schema; 
 generating a parser based on the schema using a parser algorithm associated with the schema; and 
 using the parser in a compiler to compile log files into a computer-readable format compatible with an application for identifying and evaluating cybersecurity threats from log files. 
   
     
     
         20 . The system of  claim 19 , wherein the operations further comprise:
 receiving an edited parser, wherein edits to obtain the edited parser from an initial parser include one or more of: adding code to the initial parser and removing code from the initial parser;   storing the edited parser in a repository of edited parsers associated with a user;   receiving a second log file from the user; and   parsing the second log file using the edited parser.