Log anomaly detection method and apparatus, computer device, and storage medium

Abstract

The present disclosure relates to a log anomaly detection method and apparatus, a computer device, and a storage medium. By sampling first log data to obtain a sample set of the first log data, a sample with low availability is deleted in the sampling process; the sampling of the first log data is stopped in response to the sampling completeness being high, and a sample set of the first log data is outputted; and whether second log data is anomalous data is determined based on the probability of log events from the second log data falling into the sample set of the first log data. Due to real-time updating of the sample set based on the sample availability during sampling, dynamic adaption to changes of the log data can be realized, thereby improving the anomaly detection accuracy and detection efficiency of the dynamic log data.

What is claimed is: 
     
         1 . A log anomaly detection method, comprising:
 sampling first log data to obtain a sample set of the first log data, and deleting at least one sample satisfying a first preset condition from the sample set, wherein the first preset condition is configured to represent availability of the at least one sample;   stopping the sampling of the first log data in response to the sampling of the first log data satisfying a second preset condition, and outputting the sample set of the first log data, wherein the second preset condition is configured to represent a completeness of the sampling of the first log data;   determining whether second log data is an anomalous log data based on a probability of log events contained in the second log data falling into the sample set of the first log data.   
     
     
         2 . The log anomaly detection method as claimed in  claim 1 , wherein before sampling the first log data, the method further comprises:
 tokenizing the first log data to obtain a tokenization result;   extracting key information of the first log data based on the tokenization result;   obtaining a feature vector of the first log data based on the key information.   
     
     
         3 . The log anomaly detection method as claimed in  claim 1 , wherein sampling the first log data comprises:
 selecting a first sample from the first log data;   generating a second sample based on the first sample;   calculating a probability of the first sample in a target distribution, a probability of the second sample in the target distribution, a probability of the first sample in a proposed distribution and a probability of the second sample in the proposed distribution, and determining whether the second sample is retained based on the probability of the first sample in the target distribution, the probability of the second sample in the target distribution, the probability of the first sample in the proposed distribution and the probability of the second sample in the proposed distribution, wherein the target distribution is a log data distribution corresponding to the first sample, and the proposed distribution is a log data distribution corresponding to the second sample.   
     
     
         4 . The log anomaly detection method as claimed in  claim 3 , wherein generating the second sample based on the first sample comprises:
 obtaining a Laplace distribution of the first sample;   calculating expected values of various samples relative to the first sample based on the Laplace distribution of the first sample;   generating the second sample based on the expected values.   
     
     
         5 . The log anomaly detection method as claimed in  claim 1 , wherein the first preset condition satisfies at least one of the followings: a retention time period of the sample being longer than a first preset value, and a weight of the sample being less than a second preset value. 
     
     
         6 . The log anomaly detection method as claimed in  claim 1 , wherein the second preset condition satisfies at least one of the followings: the number of iterations of the sample reaching a third preset value, a distribution of the sample reaching convergence, and a computation time period of the sample reaching a fourth preset value. 
     
     
         7 . The log anomaly detection method as claimed in  claim 1 , wherein determining whether the second log data is the anomalous log data based on the probability of log events from the second log data falling into the sample set of the first log data comprises:
 generating an anomaly score for each log event in the second log data based on the probability of log events from the second log data falling into the sample set of the first log data;   in response to the anomaly score exceeding a fifth preset value, triggering an anomaly alert, and extracting and processing the second log data triggered anomaly.   
     
     
         8 . The log anomaly detection method as claimed in  claim 5 , wherein the method further comprises:
 in response to the retention time period of the sample being longer than the first preset value, determining the at least one sample being old and configured with lower availability;   in response to the weight of the sample being less than the second preset value, determining the at least one sample configured with lower importance and availability.   
     
     
         9 . A computer device, comprising a memory, a processor, and a computer program stored in the memory and run on the processor, in response to executing the computer program, wherein the processor implements the log anomaly detection method, wherein the log anomaly detection method comprising the following steps:
 sampling first log data to obtain a sample set of the first log data, and deleting at least one sample satisfying a first preset condition from the sample set, wherein the first preset condition is configured to represent availability of the at least one sample;   stopping the sampling of the first log data in response to the sampling of the first log data satisfying a second preset condition, and outputting the sample set of the first log data, wherein the second preset condition is configured to represent a completeness of the sampling of the first log data;   determining whether second log data is an anomalous log data based on a probability of log events contained in the second log data falling into the sample set of the first log data.   
     
     
         10 . A non-transitory storage medium, wherein the non-transitory storage medium is configured to store a computer program, and in response to executed by a processor, the computer program implements the log anomaly detection method, wherein the log anomaly detection method comprising the following steps:
 sampling first log data to obtain a sample set of the first log data, and deleting at least one sample satisfying a first preset condition from the sample set, wherein the first preset condition is configured to represent availability of the at least one sample;   stopping the sampling of the first log data in response to the sampling of the first log data satisfying a second preset condition, and outputting the sample set of the first log data, wherein the second preset condition is configured to represent a completeness of the sampling of the first log data;   determining whether second log data is an anomalous log data based on a probability of log events contained in the second log data falling into the sample set of the first log data.