Training and implementing a steady state log analyzer

Abstract

The present disclosure relates to methods, systems, and computer readable media for analyzing log files for a wide variety of services (e.g., cloud computing services or microservices) to determine whether the services are operating as designed over some period of time associated with the log file(s). The present disclosure includes features and functionality for training or otherwise generating a model being configured to predict whether portions of an input log file include data reflective of normal operations of a corresponding service used to generate the input log file. The present disclosure provides a domain-agnostic approach to training an outlier detection model to analyze log files for a wide variety of services.

What is claimed is: 
     
         1 . A method, comprising:
 identifying a steady state log file for an application associated with normal operation of the application over a period of time;   applying an encoding model to the steady state log file to generate a multi-dimensional representation of the steady state log file;   generating an outlier detection model trained to determine a plurality of outlier scores for a plurality of lines of a given log file based on the multi-dimensional representation of the steady state log file being associated with normal operation of the application, wherein an outlier score indicates a predicted probability that a given line from the plurality of lines of the given log file is an outlier from normal operation of the application; and   applying the outlier detection model to an input log file to generate a plurality of outputs indicating lines of the input log file that are predicted to be outliers from normal operation of the application.   
     
     
         2 . The method of  claim 1 , wherein applying the encoding model to the steady state log file further includes:
 encoding the steady state log file to be a matrix representation of the steady state log file, the matrix representation having a same dimensionality as the steady state log file; and   applying an autoencoder to the matrix representation of the steady state log file to reduce a dimensionality of the matrix representation to a target dimensionality of the multi-dimensional representation.   
     
     
         3 . The method of  claim 2 , wherein the autoencoder reduces dimensionality of the matrix representation using a principal component analysis (PCA) engine. 
     
     
         4 . The method of  claim 1 , wherein the multi-dimensional representation includes fewer dimensions than a number of columns of the steady state log file. 
     
     
         5 . The method of  claim 1 , wherein the multi-dimensional representation of the steady state log file is a two-dimensional representation of the steady state log file. 
     
     
         6 . The method of  claim 1 , wherein the outlier detection model is a machine learning model trained to determine whether the given log file is associated with normal behavior of the application based on the multi-dimensional representation of the steady state log file. 
     
     
         7 . The method of  claim 1 , wherein the input log file is generated by the application over a second period of time with an unknown level of performance. 
     
     
         8 . The method of  claim 1 , wherein the input log file is generated by a same type of application as the application associated with the steady state log file. 
     
     
         9 . The method of  claim 1 , wherein the plurality of outputs includes a subset of lines from the input log file that are predicted to be outliers. 
     
     
         10 . The method of  claim 1 , wherein the plurality of outputs includes a set of rankings for a predetermined number of lines with highest scores associated with a high likelihood of associated lines from the input log file being outliers from normal operation of the application. 
     
     
         11 . The method of  claim 1 , wherein the multi-dimensional representation including a plurality of points representative of lines of the steady state log file within a multi-dimensional space. 
     
     
         12 . The method of  claim 11 , wherein the outlier detection model includes a defined region of the multi-dimensional space associated with normal operation of the application based on locations of the plurality of points from the multi-dimensional representation of the steady state log file within the multi-dimensional space. 
     
     
         13 . A system, comprising:
 at least one processor;   memory in electronic communication with the at least one processor; and   instructions stored in the memory, the instructions being executable by the at least one processor to:
 identify a steady state log file for an application associated with normal operation of the application over a period of time; 
 applying an encoding model to the steady state log file to generate a multi-dimensional representation of the steady state log file; 
 generating an outlier detection model trained to determine a plurality of outlier scores for a plurality of lines of a given log file based on the multi-dimensional representation of the steady state log file being associated with normal operation of the application, wherein an outlier score indicates a predicted probability that a given line from the plurality of lines of the given log file is an outlier from normal operation of the application; and 
 applying the outlier detection model to an input log file to generate a plurality of outputs indicating lines of the input log file that are predicted to be outliers from normal operation of the application. 
   
     
     
         14 . The system of  claim 13 , wherein applying the encoding model to the steady state log file further includes:
 encoding the steady state log file to be a matrix representation of the steady state log file, the matrix representation having a same dimensionality as the steady state log file; and   applying an autoencoder to the matrix representation of the steady state log file to reduce a dimensionality of the matrix representation to a target dimensionality of the multi-dimensional representation.   
     
     
         15 . The system of  claim 13 , wherein the multi-dimensional representation includes fewer dimensions than a number of columns of the steady state log file. 
     
     
         16 . The system of  claim 13 , wherein the outlier detection model is a machine learning model trained to determine whether the given log file is associated with normal behavior of the application based on the multi-dimensional representation of the steady state log file. 
     
     
         17 . The system of  claim 13 , wherein the plurality of outputs includes a subset of lines from the input log file that are predicted to be outliers. 
     
     
         18 . The system of  claim 13 , wherein the plurality of outputs includes a subset of lines from the input log file that are predicted to be outliers. 
     
     
         19 . A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, causes a computing device to:
 identify a steady state log file for an application associated with normal operation of the application over a period of time;   applying an encoding model to the steady state log file to generate a multi-dimensional representation of the steady state log file;   generating an outlier detection model trained to determine a plurality of outlier scores for a plurality of lines of a given log file based on the multi-dimensional representation of the steady state log file being associated with normal operation of the application, wherein an outlier score indicates a predicted probability that a given line from the plurality of lines of the given log file is an outlier from normal operation of the application; and   applying the outlier detection model to an input log file to generate a plurality of outputs indicating lines of the input log file that are predicted to be outliers from normal operation of the application.   
     
     
         20 . The non-transitory computer readable medium of  claim 19 , wherein applying the encoding model to the steady state log file further includes:
 encoding the steady state log file to be a matrix representation of the steady state log file, the matrix representation having a same dimensionality as the steady state log file; and   applying an autoencoder to the matrix representation of the steady state log file to reduce a dimensionality of the matrix representation to a target dimensionality of the multi-dimensional representation.