Method and system for detecting threats using passive cluster mapping

Abstract

An approach for detecting network threats is disclosed, that may involve receiving network traffic, plotting the network traffic in a n-dimensional feature space to form a network map, generating a client signature at least by placing new client points in the map, setting a threshold, and generating an alarm if one or more client activity points exceed the threshold. In some embodiments, the network map and the client signature are updated using sliding windows and distance calculations.

What is claimed is: 
     
       1. A method for detecting network threats, comprising:
 receiving network traffic by tapping a network device that routes network communications generated by a plurality of clients in a network; 
 mapping the network traffic in a network feature space as a first set of client points; 
 forming client groups from the first set of clients points, wherein points in a client group share a center point; 
 mapping distances from a second set of client points to the client groups in the network feature space; 
 generating client signature data corresponding to a plot, the plot comprising a first axis corresponding to a distance between the second set of client points and one or more client groups and a second axis associated with time, the second axis comprising of a first window and a second window, wherein the client signature data comprises a series of client points that are added to a first window along a first axis, the series of client point comprising a height along a second axis, the height corresponding to the distance from an individual client point from among the series of client points to the one or more of the client groups, wherein a portion of the client signature data corresponds to a sliding window; 
 initiating a detection phase that identifies abnormal network behavior by:
 inputting one or more client activity events from the network traffic along the first axis in the second window, and 
 generating a dynamic threshold based at least on aggregated client activity points from at least the first window, wherein the dynamic threshold corresponding to a set distance from the first axis at least in the second window is updated in response to new client activity points entering a sliding window; 
 
 tuning the dynamic threshold to change a strictness policy to match individual networks; and 
 generating alarm data in response to the one or more client activity events exceeding the dynamic threshold, wherein the alarm data indicates a threat detection. 
 
     
     
       2. The method of  claim 1 , wherein the dynamic threshold is adjustable as a function of the client signature data. 
     
     
       3. The method of  claim 2 , wherein the function applies one or more of following to the client signature data: averaging, normalization through mean, or standard deviation, and a multiplication factor. 
     
     
       4. The method of  claim 1 , wherein a client point in the client signature data is computed by averaging the distances from the client point to the client groups. 
     
     
       5. The method of  claim 1 , wherein new points are added to the first set of client points, wherein individual points of the new points are grouped together with a respective closest client group. 
     
     
       6. The method of  claim 1 , wherein the closest client group for an individual point is determined by measuring the distance between the individual point and a centroid of the closest client group. 
     
     
       7. The method of  claim 1 , wherein the network feature space is an n-dimensional feature space having one or more axes that correspond to different types of network features and the first set of client points are mapped in the network feature space according to their respective values in relation to the one or more axes. 
     
     
       8. The method of  claim 7 , wherein the one or more axes correspond to one or more of the following features: a client identifier, a server identifier, a control module identifier, account information, services accessed, and/or a number of times a service is accessed. 
     
     
       9. A system for detecting network threats, comprising:
 a computer processor to execute a set of program code instructions; 
 
       a memory to hold the set of program code instructions, in which the set of program code instructions comprises program code to perform:
 receive network traffic by tapping a network device that routes network communications generated by a plurality of clients in a network; 
 map the network traffic in a network feature space as a first set of client points; 
 form client groups from the first set of clients points, wherein points in a client group share a center point; map distances from a second set of client points to the client groups in the network feature space; 
 generate client signature data corresponding to a plot, the plot comprising a first axis corresponding to a distance between the second set of client points and one or more client groups and a second axis associated with time, the second axis comprising of a first window and a second window, wherein the client signature data comprises a series of client points that are added to a first window along a first axis, the series of client point comprising a height along a second axis, the height corresponding to the distance from an individual client point from among the series of client points to the one or more of the client groups, wherein a portion of the client signature data corresponds to a sliding window; 
 initiate a detection phase that identifies abnormal network behavior by:
 inputting one or more client activity events from the network traffic along the first axis in the second window, and 
 generating a dynamic threshold based at least on aggregated client activity points from at least the first window, wherein the dynamic threshold corresponding to a set distance from the first axis at least in the second window is updated in response to new client activity points entering a sliding window; 
 
 tuning the dynamic threshold to change a strictness policy to match individual networks; and 
 generate alarm data in response to one or more client activity events exceeding the dynamic threshold, wherein the alarm data indicates a threat detection. 
 
     
     
       10. The system of  claim 9 , wherein the dynamic threshold is adjustable as a function of the client signature data. 
     
     
       11. The system of  claim 10 , wherein the function applies one or more of following to the client signature data: averaging, normalization through mean, or standard deviation, and a multiplication factor. 
     
     
       12. The system of  claim 9 , wherein a client point in the client signature data is computed by averaging the distances from the client point to the client groups. 
     
     
       13. The system of  claim 9 , wherein new points are added to the first set of client points, wherein individual points of the new points are grouped together with a respective closest client group. 
     
     
       14. The system of  claim 9 , wherein the closest client group for an individual point is determined by measuring the distance between the individual point and a centroid of the closest client group. 
     
     
       15. The system of  claim 9 , wherein the network feature space is an n-dimensional feature space having one or more axes that correspond to different types of network features and the first set of client points are mapped in the network feature space according to a respective values in relation to the one or more axes. 
     
     
       16. The system of  claim 15 , wherein the one or more axes correspond to one or more of the following features: a client identifier, a server identifier, a control module identifier, account information, services accessed, and/or a number of times a service is accessed. 
     
     
       17. A computer program product embodied on a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method for detecting network threats, the method comprising:
 receiving network traffic by tapping a network device that routes network communications generated by a plurality of clients in a network; 
 mapping the network traffic in a network feature space as a first set of client points; 
 forming client groups from the first set of clients points, wherein points in a client group share a center point; 
 mapping distances from a second set of client points to the client groups in the network feature space; 
 generating client signature data corresponding to a plot, the plot comprising a first axis corresponding to a distance between the second set of client points and one or more client groups and a second axis associated with time, the second axis comprising of a first window and a second window, wherein the client signature data comprises a series of client points that are added to a first window along a first axis, the series of client point comprising a height along a second axis, the height corresponding to the distance from an individual client point from among the series of client points to the one or more of the client groups, wherein a portion of the client signature data corresponds to a sliding window; 
 initiating a detection phase that identifies abnormal network behavior by:
 inputting one or more client activity events from the network traffic along the first axis in the second window, and 
 generating a dynamic threshold based at least on aggregated client activity points from at least the first window, wherein the dynamic threshold corresponding to a set distance from the first axis at least in the second window is updated in response to new client activity points entering a sliding window; 
 
 tuning the dynamic threshold to change a strictness policy to match individual networks; and 
 generating alarm data in response to the one or more client activity events exceeding the dynamic threshold, wherein the alarm data indicates a threat detection. 
 
     
     
       18. The computer program product of  claim 17 , wherein the dynamic threshold is adjustable as a function of the client signature data. 
     
     
       19. The computer program product of  claim 18 , wherein the function applies one or more of following to the client signature data: averaging, normalization through mean, or standard deviation, and a multiplication factor. 
     
     
       20. The computer program product of  claim 17 , wherein a client point in the client signature data is computed by averaging the distances from the client point to the client groups. 
     
     
       21. The computer program product of  claim 17 , wherein new points are added to the first set of client points, wherein individual points of the new points are grouped together with a respective closest client group. 
     
     
       22. The computer program product of  claim 17 , wherein the closest client group for an individual point is determined by measuring the distance between the individual point and a centroid of the closest client group. 
     
     
       23. The computer program product of  claim 17 , wherein the network feature space is an n-dimensional feature space having one or more axes that correspond to different types of network features and the first set of client points are mapped in the network feature space according to their respective values in relation to the one or more axes. 
     
     
       24. The computer program product of  claim 23 , wherein the one or more axes correspond to one or more of the following features: a client identifier, a server identifier, a control module identifier, account information, services accessed, and/or a number of times a service is accessed.