Metric-based anomaly detection system with evolving mechanism in large-scale cloud

Abstract

A computer-implemented method is presented for detecting anomalies in dynamic datasets generated in a cloud computing environment. The method includes monitoring a plurality of cloud servers receiving a plurality of data points, employing a two-level clustering training module to generate micro-clusters from the plurality of data points, each of the micro-clusters representing a set of original data from the plurality of data points, employing a detecting module to detect normal data points, abnormal data points, and unknown data points from the plurality of data points via a detection model, employing an evolving module using a different evolving mechanism for each of the normal, abnormal, and unknown data points to evolve the detection model, and generating a system report displayed on a user interface, the system report summarizing the micro-cluster information.

What is claimed is: 
     
       1. A computer-implemented method executed on a processor for detecting anomalies in dynamic datasets generated in a cloud computing environment, the computer-implemented method comprising:
 employing a two-level clustering training module to generate micro-clusters from a plurality of data points collected from cloud servers, each of the micro-clusters representing a set of original data from the plurality of data points, and each of the micro-clusters is inspected to determine whether overlap exists among them; 
 detecting normal data points, abnormal data points, and unknown data points from the plurality of data points via a detection model; 
 evolving the detection model by a plurality of different evolving mechanisms; and 
 generating a system report displayed on a user interface, the system report summarizing the micro-cluster information. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein the plurality of different evolving mechanisms include an evolving mechanism for each of the normal, abnormal, and unknown data points. 
     
     
       3. The computer-implemented method of  claim 2 , wherein the normal points and the abnormal points are immediately evolved in the detection model and the unknown points are temporarily saved in a memory. 
     
     
       4. The computer-implemented method of  claim 1 , wherein the two-level clustering training module is trained with historical data stored in a historical information database. 
     
     
       5. The computer-implemented method of  claim 1 , wherein the generated micro-clusters are normal micro-clusters, abnormal micro-clusters, and unknown micro-clusters. 
     
     
       6. The computer-implemented method of  claim 5 , wherein the normal micro-clusters decay through time, the abnormal micro-clusters do not decay through time, and the unknown micro-clusters decay through time at a quicker rate than the normal micro-clusters. 
     
     
       7. The computer-implemented method of  claim 5 , wherein long-term unknown micro-clusters are transformed to default abnormal micro-clusters. 
     
     
       8. The computer-implemented method of  claim 5 , wherein the micro-cluster information is permitted to be modified by a user, and, when the abnormal micro-clusters are generated, the user receives a notification. 
     
     
       9. A non-transitory computer-readable storage medium comprising a computer-readable program executed on a processor in a data processing system for detecting anomalies in dynamic datasets generated in a cloud computing environment, wherein the computer-readable program when executed on the processor causes a computer to perform the steps of:
 employing a two-level clustering training module to generate micro-clusters from a plurality of data points collected from cloud servers, each of the micro-clusters representing a set of original data from the plurality of data points, and each of the micro-clusters is inspected to determine whether overlap exists among them; 
 detecting normal data points, abnormal data points, and unknown data points from the plurality of data points via a detection model; 
 evolving the detection model by a plurality of different evolving mechanisms; and 
 generating a system report displayed on a user interface, the system report summarizing the micro-cluster information. 
 
     
     
       10. The non-transitory computer-readable storage medium of  claim 9 , wherein the plurality of different evolving mechanisms include an evolving mechanism for each of the normal, abnormal, and unknown data points. 
     
     
       11. The non-transitory computer-readable storage medium of  claim 10 , wherein the normal points and the abnormal points are immediately evolved in the detection model and the unknown points are temporarily saved in a memory. 
     
     
       12. The non-transitory computer-readable storage medium of  claim 9 , wherein the two-level clustering training module is trained with historical data stored in a historical information database. 
     
     
       13. The non-transitory computer-readable storage medium of  claim 9 , wherein the generated micro-clusters are normal micro-clusters, abnormal micro-clusters, and unknown micro-clusters. 
     
     
       14. The non-transitory computer-readable storage medium of  claim 13 , wherein the normal micro-clusters decay through time, the abnormal micro-clusters do not decay through time, and the unknown micro-clusters decay through time at a quicker rate than the normal micro-clusters. 
     
     
       15. The non-transitory computer-readable storage medium of  claim 13 , wherein long-term unknown micro-clusters are transformed to default abnormal micro-clusters. 
     
     
       16. The non-transitory computer-readable storage medium of  claim 13 , wherein the micro-cluster information is permitted to be modified by a user, and, when the abnormal micro-clusters are generated, the user receives a notification. 
     
     
       17. A system for detecting anomalies in dynamic datasets generated in a cloud computing environment, the system comprising:
 a two-level clustering training component generates micro-clusters from a plurality of data points received from a plurality of cloud servers, each of the micro-clusters representing a set of original data from the plurality of data points, and each of the micro-clusters is inspected to determine whether overlap exists among them; 
 a detector detects normal data points, abnormal data points, and unknown data points from the plurality of data points via a detection model; and 
 a system report generated to be displayed on a user interface, the system report summarizing the micro-cluster information, 
 wherein normal micro-clusters of the generated micro-clusters decay through time and abnormal micro-clusters of the generated micro-clusters do not decay through time. 
 
     
     
       18. The system of  claim 17 , wherein an evolving component uses a different evolving mechanism for each of the normal, abnormal, and unknown data points to evolve the detection model. 
     
     
       19. The system of  claim 17 , wherein the two-level clustering training module is trained with historical data stored in a historical information database. 
     
     
       20. The system of  claim 17 ,
 wherein unknown micro-clusters of the generated micro-clusters decay through time at a quicker rate than the normal micro-clusters.