Anomaly detection with feedback

Abstract

Examples of techniques for anomaly detection with feedback are described. An instance includes a technique is receiving a plurality of unlabeled data points from an input stream; performing anomaly detection on a point of the unlabeled data points using an anomaly detection engine; pre-processing the unlabeled data point that was subjected to anomaly detection; classifying the pre-processed unlabeled data point; determining the anomaly detection was not proper based on a comparison of a result of the anomaly detection and a result of the classifying of the pre-processed unlabeled data point; and in response to determining the anomaly detection was not proper, providing feedback to the anomaly detection engine to change at least one emphasis used in anomaly detection.

What is claimed is: 
     
       1. An apparatus comprising:
 one or more non-transitory computer-readable storage media; and 
 program instructions stored on the one or more non-transitory computer readable storage media that, when read and executed by a processing system, direct the processing system to: 
 receive a plurality of unlabeled data points from an input stream; 
 perform anomaly detection on a data point of the unlabeled data points using an anomaly detection engine; 
 pre-process the data point of the unlabeled data points to generate a pre-processed unlabeled data point; and 
 classify the pre-processed unlabeled data point by:
 maintaining, in memory, a graph of nodes including a defined maximum dimension, such that when a first node is removed from the graph of nodes the first node is replaced with a second node including a same amount of information as the first node, such that the memory maintains the defined maximum dimension of the graph of nodes; 
 removing an oldest node from the graph of nodes; and 
 inserting a new node in the graph of nodes for the pre-processed unlabeled data point, the new node including a same amount of information as the oldest node, such that the memory maintains the defined maximum dimension of the graph of nodes. 
 
 
     
     
       2. The apparatus of  claim 1 , wherein the pre-processing is performed by generating a normalized shingle difference between a shingle including the anomalous data points and subsequent shingles of unlabeled data points in the input stream. 
     
     
       3. The apparatus of  claim 1 , wherein the anomaly detection comprises a robust random cut forest detection. 
     
     
       4. A computer-implemented method comprising:
 receiving a plurality of unlabeled data points from an input stream; 
 performing anomaly detection on a data point of the unlabeled data points using an anomaly detection engine; 
 pre-processing the data point of the unlabeled data points to generate a pre-processed unlabeled data point; and 
 classifying the pre-processed unlabeled data point by:
 maintaining, in memory, a graph of nodes including a defined maximum dimension, such that when a first node is removed from the graph of nodes the first node is replaced with a second node including a same amount of information as the first node, such that the memory maintains the defined maximum dimension of the graph of nodes; 
 removing an oldest node from the graph of nodes; and 
 inserting a new node in the graph of nodes for the pre-processed unlabeled data point, the new node including a same amount of information as the oldest node, such that the memory maintains the defined maximum dimension of the graph of nodes. 
 
 
     
     
       5. The computer-implemented method of  claim 4 , wherein the pre-processing the data point of the unlabeled data points comprises generating a normalized shingle difference between a shingle including the data point of the unlabeled data points and subsequent shingles of unlabeled data points in the input stream. 
     
     
       6. The computer-implemented method of  claim 4 , wherein the input stream is a real-time stream. 
     
     
       7. The computer-implemented method of  claim 4 , wherein the classifying the pre-processed unlabeled data point further comprises updating edge weights associated with weighted edges between the nodes in the graph of nodes. 
     
     
       8. The computer-implemented method of  claim 4 , wherein removing the oldest node from the graph of nodes comprises star-meshing out the oldest node from the graph of nodes. 
     
     
       9. The computer-implemented method of  claim 8 , wherein the graph of nodes is stored in the memory as a matrix. 
     
     
       10. The computer-implemented method of  claim 9 , wherein the matrix is a Laplacian matrix. 
     
     
       11. The computer-implemented method of  claim 10 , wherein the classifying the pre-processed unlabeled data point further comprises computing fractional labels for unlabeled nodes of the graph of nodes using a pseudo-inverse of the Laplacian matrix. 
     
     
       12. The computer-implemented method of  claim 4 , wherein the anomaly detection comprises a robust random cut forest detection. 
     
     
       13. The computer-implemented method of  claim 4 , further comprising outputting information about the pre-processed unlabeled data point to a feedback generator. 
     
     
       14. A system comprising:
 a processor; and 
 a memory storing instructions that, when executed by the processor, cause the processor to:
 receive a plurality of unlabeled data points from an input stream; 
 perform anomaly detection on a data point of the unlabeled data points using an anomaly detection engine; 
 pre-process the data point of the unlabeled data points to generate a pre-processed unlabeled data point; 
 classify the pre-processed unlabeled data point by:
 maintaining, in the memory, a graph of nodes including a defined maximum dimension, such that when a first node is removed from the graph of nodes the first node is replaced with a second node including a same amount of information as the first node, such that the memory maintains the defined maximum dimension of the graph of nodes; 
 removing an oldest node from the graph of nodes; and 
 inserting a new node in the graph of nodes for the pre-processed unlabeled data point, the new node including a same amount of information as the oldest node, such that the memory maintains the defined maximum dimension of the graph of nodes; 
 
 determine the anomaly detection was not proper based on a comparison of a result of the anomaly detection and a result of the classifying of the pre-processed unlabeled data point; and 
 in response to determining the anomaly detection was not proper, provide feedback to the anomaly detection engine to change at least one emphasis used in the anomaly detection. 
 
 
     
     
       15. The system of  claim 14 , wherein the pre-processing the data point of the unlabeled data points comprises generating a normalized shingle difference between a shingle including the data point of the unlabeled data points and subsequent shingles of unlabeled data points in the input stream. 
     
     
       16. The system of  claim 14 , wherein the input stream is a real-time stream. 
     
     
       17. The system of  claim 14 , wherein the classifying the pre-processed unlabeled data point further comprises updating edge weights associated with weighted edges between the nodes in the graph of nodes. 
     
     
       18. The system of  claim 14 , wherein removing the oldest node from the graph of nodes comprises star-meshing out the oldest node from the graph of nodes. 
     
     
       19. The system of  claim 14 , wherein the graph of nodes is stored in the memory as a matrix. 
     
     
       20. The system of  claim 19 , wherein the matrix is a Laplacian matrix. 
     
     
       21. The system of  claim 20 , wherein the classifying the pre-processed unlabeled data point further comprises computing fractional labels for unlabeled nodes of the graph of nodes using a pseudo-inverse of the Laplacian matrix.