Anomaly and Causation Detection in Computing Environments Using Counterfactual Processing

Abstract

Anomaly and causation detection in computing environments are disclosed. An example method includes receiving an input stream of data instances for a time series, each of the data instances being time stamped and including at least one principle value and a set of categorical attributes; generating anomaly scores for each of the data instances over time intervals; detecting a change in the anomaly scores over the time intervals for the data instances; and identifying which of the set of categorical attributes of the data instances caused the change in the anomaly scores using a counterfactual analysis. The counterfactual analysis may comprise removing a portion of the data instances; regenerating the anomaly scores for each of the remaining data instances over the time intervals; and if the anomaly scores are improved, identifying the portion as a cause of anomalous activity. Recommendations to remediate the cause may be generated.

What is claimed is: 
     
         1 . A method for detecting anomalous activity in a computing environment, the method comprising:
 receiving an input stream of data instances for a time series, where the data instances in the input stream are time stamped, each of the data instances comprising at least one principle value and a set of categorical attributes;   generating anomaly scores for each of the data instances over time intervals;   detecting a change in the anomaly scores over the time intervals for the data instances; and   identifying which of the set of categorical attributes of the data instances caused the change in the anomaly scores using a counterfactual analysis, the counterfactual analysis comprising:
 removing at least a portion of the data instances; 
 regenerating the anomaly scores for each of the data instances over the time intervals; and 
 wherein if the regenerated anomaly scores are improved compared to the anomaly scores, at least a portion of the categorical attributes are identified as anomalous categorical attributes and a cause of the anomalous activity. 
   
     
     
         2 . The method of  claim 1 , wherein the change in the anomaly scores is indicative of malicious behavior in the computing environment. 
     
     
         3 . The method of  claim 2 , further comprising generating recommendations for remediating the set of categorical attributes to remediate the malicious behavior. 
     
     
         4 . The method of  claim 3 , wherein at least one of the generated recommendations comprises a recommendation that all devices accessing a database use a higher level of authentication with respect to the database. 
     
     
         5 . The method of  claim 1 , wherein the set of categorical attributes comprises a tuple created from at least two categorical attributes. 
     
     
         6 . The method of  claim 1 , further comprising grouping the data instances into groups based on the time intervals, each of the groups having a time length for its corresponding time interval. 
     
     
         7 . The method of  claim 1 , wherein the at least one principle value is categorical or numerical. 
     
     
         8 . The method of  claim 7 , wherein, for the numerical principle value, a set function is applied to calculate a mean value. 
     
     
         9 . The method of  claim 7 , wherein, for the categorical principle value, a set function is applied to calculate any of an equivalence class count or a distinct count. 
     
     
         10 . The method of  claim 1 , wherein generating the anomaly scores comprises:
 creating features for a current group of the data instances;   applying an anomaly detection algorithm that takes as inputs the features for the current group, and group features calculated using set functions for groups earlier than the current group; and   generating the anomaly scores, the anomaly scores being indicative of how anomalous are the features for the current group.   
     
     
         11 . The method of  claim 1 , further comprising enacting changes in the computing environment relative to at least a portion of the categorical attributes to prevent future instances of the anomalous activity. 
     
     
         12 . A method for detecting anomalous activity in a computing environment, the method comprising:
 receiving an input stream of data instances, the data instances in the input stream being time stamped;   separating the data instances into at least one principle value and a set of categorical attributes;   grouping the data instances into groups based on time intervals, each of the time intervals having a length;   applying set functions to each of the groups;   generating an anomaly score for each of the groups using the set functions; and   applying a counterfactual analysis or a regularity analysis to identify which of the set of categorical attributes for a group is influencing one or more anomalies in the groups that are indicative of the anomalous activity in the computing environment, wherein the counterfactual analysis comprises:
 determining a change in the anomaly score; 
 removing at least a portion of the data instances, the at least a portion of the data instances being associated with one or more categorical attributes of the set of categorical attributes identified as influencing the one or more anomalies; 
 regenerating the anomaly score for each of the data instances which remain after the removing; and 
 comparing the regenerated anomaly score to the anomaly score to identify if at least a portion of the categorical attributes caused the change in the anomaly score. 
   
     
     
         13 . The method of  claim 12 , further comprising remediating the computing environment to remedy the anomalous activity. 
     
     
         14 . The method of  claim 12 , wherein generating the anomaly score further comprises applying an anomaly detection algorithm to values generated using the set function to detect changes in the groups over the time intervals. 
     
     
         15 . The method of  claim 12 , further comprising generating recommendations for remediating the set of categorical attributes to remediate the anomalous activity. 
     
     
         16 . The method of  claim 12 , wherein the regularity analysis further comprises identifying when a categorical attribute of the set of categorical attributes influences the anomaly score for the set of categorical attributes if an output of an anomaly detection algorithm is approximately identical to alternative instances in which the set of categorical attributes exists. 
     
     
         17 . A system for detecting anomalous activity in a computing environment, comprising:
 a processor; and   a memory for storing executable instructions, the processor executing the instructions to perform an unsupervised machine learning method that comprises:
 generating anomaly scores for data instances of an input stream received over time intervals; 
 detecting a change in the anomaly scores over the time intervals for the data instances; and 
 identifying which of a set of categorical attributes of the data instances caused the anomaly scores using a counterfactual analysis or a regularity analysis, wherein the counterfactual analysis comprises:
 removing at least a portion of the data instances; 
 regenerating the anomaly scores for each of the data instances, that remained after the removing, over the time intervals; and 
 wherein if the regenerated anomaly scores are improved compared to the anomaly scores, at least a portion of the categorical attributes are identified as anomalous categorical attributes and a cause of the anomalous activity. 
 
   
     
     
         18 . The system of  claim 17 , wherein the data instances correspond to selected features to be analyzed for anomalous behavior. 
     
     
         19 . The system of  claim 17 , further comprising remediating the computing environment to remedy the anomalous activity associated with the anomalous behavior. 
     
     
         20 . The system of  claim 17 , further comprising generating recommendations for remediating the set of categorical attributes to remediate the anomalous activity associated with the anomalous behavior.