Method and apparatus for identifying and detecting threats to an enterprise or e-commerce system

Abstract

Methods and apparatuses for identifying and detecting threats to an enterprise or e-commerce system are disclosed, including grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system; extracting one or more features from the grouped log lines into one or more features tables; using one or more statistical models on the one or more features tables to identify statistical outliers; labeling the statistical outliers to create one or more labeled features tables; using the one or more labeled features tables to create one or more rules for identifying threats to the enterprise or e-commerce system; and using the one or more rules on incoming enterprise or e-commerce system data traffic to detect threats to the enterprise or e-commerce system. Other embodiments are described and claimed.

1 . A method for identifying and detecting threats to an enterprise or e-commerce system, the method comprising:
 grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system;   extracting one or more features from the grouped log lines into one or more features tables;   using one or more statistical models on the one or more features tables to identify statistical outliers;   labeling the statistical outliers to create one or more labeled features tables; and   using the one or more labeled features tables to create one or more rules for identifying threats to the enterprise or e-commerce system.   
     
     
         2 . The method of  claim 1 , wherein the method further comprises ranking and rearranging the one or more log line parameters of the one or more features tables by probability. 
     
     
         3 . The method of  claim 1 , the method further comprising using the one or more rules on the incoming data traffic to the enterprise or e-commerce system to detect threats to the enterprise or e-commerce system. 
     
     
         4 . The method of  claim 2 , wherein the using the one or more rules on the incoming data traffic to the enterprise or e-commerce system to detect threats to the enterprise or e-commerce system is in real-time. 
     
     
         5 . The method of  claim 2 , the method further comprising using the detected threats to modify the one or more statistical models and/or the one or more rules for identifying threats to the enterprise or e-commerce system. 
     
     
         6 . The method of  claim 2 , the method further comprising blocking and/or challenging the packet flow of the detected threats. 
     
     
         7 . The method of  claim 1 , wherein the one or more log line parameters comprises at least one of: user ID, session, IP address, and URL query. 
     
     
         8 . The method of  claim 1 , wherein the one or more enterprise or e-commerce system data sources comprises at least one of: web server access logs, firewall logs, packet captures per application, active directory logs, DNS logs, forward proxy logs, external threat feeds, AV logs, user logon audits, DLP logs, LB logs, IPS/IDS logs, black listed URLs, black listed IP addresses, and black listed referrers. 
     
     
         9 . The method of  claim 1 , wherein the one or more features comprises at least one of: user session duration, length of user URL query, number of characters of user URL query, number of digits of user URL query, number of punctuations of user URL query, number of requests in user session, average time between clicks in user session, user session click rate, percentage of image requests in user session, percentage of 4xx responses in user session, percentage of 3xx responses in user session, percentage of 2xx responses in user session, percentage of zip responses in user session, percentage of binary responses in user session, percentage of head requests in user session, number of checkouts, number of credit cards added, number of promo codes added, number of gift cards added, number of times items were shipped overnight, number of times new shipping address was added, number of login failures, number of login successes, number of password resets, and total number of requests. 
     
     
         10 . The method of  claim 1 , wherein the one or more statistical models comprises at least one of: Clustering models, Hidden Markov model, and Copula models. 
     
     
         11 . The method of  claim 1 , wherein the one or more rules for identifying threats to the enterprise or e-commerce system comprises a random forest classifier, learning vector quantization, and/or a neural network. 
     
     
         12 . The method of  claim 1 , wherein the using one or more statistical models on the one or more features comprises using a bag of little bootstraps sampling. 
     
     
         13 . The method of  claim 1 , wherein the using one or more statistical models on the one or more features tables to identify statistical outliers comprises:
 distributing one or more features from the one or more features tables across two or more servers;   using the one or more statistical models on the distributed one or more features; and   aggregating results from the using the one or more statistical models on the distributed one or more features.   
     
     
         14 . The method of  claim 13 , wherein the using the one or more statistical models on the distributed one or more features comprises using a bag of little bootstraps sampling. 
     
     
         15 . The method of  claim 1 , wherein the labeling the statistical outliers to create one or more labeled features tables comprises presenting an administrator the statistical outliers for identification as malicious, non-malicious, or other administrator defined label. 
     
     
         16 . An apparatus for identifying and detecting threats to an enterprise or e-commerce system, the apparatus comprising:
 one or more processors;   system memory coupled to the one or more processors;   one or more non-transitory memory units coupled to the one or more processors; and   threat identification and detection code stored on the one or more non-transitory memory units that when executed by the one or more processors are configured to perform a method, comprising:
 grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system; 
 extracting one or more features from the grouped log lines into one or more features tables; 
 using one or more statistical models on the one or more features tables to identify statistical outliers; 
 labeling the statistical outliers to create one or more labeled features tables; and 
 using the one or more labeled features tables to create one or more rules for identifying threats to the enterprise or e-commerce system. 
   
     
     
         17 . The apparatus of  claim 16 , wherein the method further comprises ranking and rearranging the one or more log line parameters of the one or more features tables by probability. 
     
     
         18 . The apparatus of  claim 16 , wherein the method further comprises using the one or more rules on the incoming data traffic to the enterprise or e-commerce system to detect threats to the enterprise or e-commerce system. 
     
     
         19 . The apparatus of  claim 17 , wherein the using the one or more rules on the incoming data traffic to the enterprise or e-commerce system to detect threats to the enterprise or e-commerce system is in real-time. 
     
     
         20 . The apparatus of  claim 17 , wherein the method further comprises using the detected threats to modify the one or more statistical models and/or the one or more rules for identifying threats to the enterprise or e-commerce system. 
     
     
         21 . The apparatus of  claim 17 , wherein the method further comprises blocking and/or challenging the packet flow of the detected threats. 
     
     
         22 . The apparatus of  claim 16 , wherein the one or more log line parameters comprises at least one of: user ID, session, IP address, and URL query. 
     
     
         23 . The apparatus of  claim 16 , wherein the one or more enterprise or e-commerce system data sources comprises at least one of: web server access logs, firewall logs, packet captures per application, active directory logs, DNS logs, forward proxy logs, external threat feeds, AV logs, user logon audits, DLP logs, LB logs, IPS/IDS logs, black listed URLs, black listed IP addresses, and black listed referrers. 
     
     
         24 . The apparatus of  claim 16 , wherein the one or more features comprises at least one of: user session duration, length of user URL query, number of characters of user URL query, number of digits of user URL query, number of punctuations of user URL query, number of requests in user session, average time between clicks in user session, user session click rate, percentage of image requests in user session, percentage of 4xx responses in user session, percentage of 3xx responses in user session, percentage of 2xx responses in user session, percentage of zip responses in user session, percentage of binary responses in user session, percentage of head requests in user session, number of checkouts, number of credit cards added, number of promo codes added, number of gift cards added, number of times items were shipped overnight, number of times new shipping address was added, number of login failures, number of login successes, number of password resets, and total number of requests. 
     
     
         25 . The apparatus of  claim 16 , wherein the one or more statistical models comprises at least one of: Clustering models, Hidden Markov model, and Copula models. 
     
     
         26 . The apparatus of  claim 16 , wherein the one or more rules for identifying threats to the enterprise or e-commerce system comprises a random forest classifier, learning vector quantization, and/or a neural network. 
     
     
         27 . The apparatus of  claim 16 , wherein the using one or more statistical models on the one or more features comprises using a bag of little bootstraps sampling. 
     
     
         28 . The apparatus of  claim 16 , wherein the using one or more statistical models on the one or more features tables to identify statistical outliers comprises:
 distributing one or more features from the one or more features tables across two or more servers;   using the one or more statistical models on the distributed one or more features; and   aggregating results from the using the one or more statistical models on the distributed one or more features.   
     
     
         29 . The apparatus of  claim 28 , wherein the using the one or more statistical models on the distributed one or more features comprises using a bag of little bootstraps sampling. 
     
     
         30 . The apparatus of  claim 16 , wherein the labeling the statistical outliers to create one or more labeled features tables comprises presenting an administrator the statistical outliers for identification as malicious, non-malicious, or other administrator defined label. 
     
     
         31 . An apparatus for identifying and detecting threats to an enterprise or e-commerce system, the apparatus comprising:
 a pattern discoverer; and   one or more pattern normalizers coupled to the pattern discover;   wherein at least one of the one or more pattern normalizers comprise:
 one or more pattern normalizer processors; 
 pattern normalizer system memory coupled to the one or more pattern normalizer processors; 
 one or more pattern normalizer non-transitory memory units coupled to the one or more pattern normalizer processors; 
 a pattern normalizer communications device coupled to the one or more pattern normalizer processors, the pattern normalizer communications device being configured to communicate with the pattern discover; and 
 pattern normalizer code stored on the one or more pattern normalizer non-transitory memory units that when executed by the one or more pattern normalizer processors are configured to perform a pattern normalizer method, comprising:
 grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system; 
 extracting one or more features from the grouped log lines into one or more features tables; and 
 sending the one or more features tables to the pattern discoverer; and 
 
   wherein the pattern discoverer comprises:
 one or more pattern discoverer processors; 
 pattern discoverer system memory coupled to the one or more pattern discoverer processors; 
 one or more pattern discoverer non-transitory memory units coupled to the one or more pattern discoverer processors; 
 a pattern discoverer communications device coupled to the one or more pattern discoverer processors, the pattern discoverer communications device being configured to communicate with the one or more pattern normalizers; and 
 pattern discoverer code stored on the one or more pattern discoverer non-transitory memory units that when executed by the one or more pattern discoverer processors are configured to perform a pattern discoverer method, comprising:
 using one or more statistical models on the one or more features tables to identify statistical outliers; 
 labeling the statistical outliers to create one or more labeled features tables; and 
 using the one or more labeled features tables to create one or more rules for identifying threats to the enterprise or e-commerce system. 
 
   
     
     
         32 . The apparatus of  claim 31 , wherein the pattern discoverer method further comprises ranking and rearranging the one or more log line parameters of the one or more features tables by probability. 
     
     
         33 . The apparatus of  claim 31 , the apparatus further comprising one or more threat detectors coupled to the pattern discover, wherein the pattern discoverer method further comprises sending to the one or more threat detectors, the one or more rules for identifying threats to the enterprise or e-commerce system; and wherein at least one of the one or more threat detectors comprise:
 one or more threat detector processors;   threat detector system memory coupled to the one or more threat detector processors;   one or more threat detector non-transitory memory units coupled to the one or more threat detector processors;   a threat detector communications device coupled to the one or more threat detector processors, the threat detector communications device being configured to communicate with the pattern discover; and   threat detector code stored on the one or more threat detector non-transitory memory units that when executed by the one or more threat detector processors are configured to perform a threat detector method, comprising:
 using the one or more rules on the incoming data traffic to the enterprise or e-commerce system to detect threats to the enterprise or e-commerce system. 
   
     
     
         34 . The apparatus of  claim 32 , wherein the using the one or more rules on the incoming data traffic to the enterprise or e-commerce system to detect threats to the enterprise or e-commerce system is in real-time. 
     
     
         35 . The apparatus of  claim 32 , wherein the threat detector method further comprises sending the detected threats to the pattern discoverer; and wherein the pattern discoverer method further comprises using the detected threats to modify the one or more statistical models and/or the one or more rules for identifying threats to the enterprise or e-commerce system. 
     
     
         36 . The apparatus of  claim 32 , wherein the threat detector method further comprises blocking and/or challenging the packet flow of the detected threats. 
     
     
         37 . The apparatus of  claim 31 , wherein the one or more log line parameters comprises at least one of: user ID, session, IP address, and URL query. 
     
     
         38 . The apparatus of  claim 31 , the apparatus further comprising a cloud server linked to the pattern discoverer, the cloud server being configured to share the one or more rules for identifying threats to the enterprise or e-commerce system with one or more enterprise or e-commerce systems. 
     
     
         39 . The apparatus of  claim 31 , wherein the one or more enterprise or e-commerce system data sources comprises at least one of: web server access logs, firewall logs, packet captures per application, active directory logs, DNS logs, forward proxy logs, external threat feeds, AV logs, user logon audits, DLP logs, LB logs, IPS/IDS logs, black listed URLs, black listed IP addresses, and black listed referrers. 
     
     
         40 . The apparatus of  claim 31 , wherein the one or more features comprises at least one of: user session duration, length of user URL query, number of characters of user URL query, number of digits of user URL query, number of punctuations of user URL query, number of requests in user session, average time between clicks in user session, user session click rate, percentage of image requests in user session, percentage of 4xx responses in user session, percentage of 3xx responses in user session, percentage of 2xx responses in user session, percentage of zip responses in user session, percentage of binary responses in user session, percentage of head requests in user session, number of checkouts, number of credit cards added, number of promo codes added, number of gift cards added, number of times items were shipped overnight, number of times new shipping address was added, number of login failures, number of login successes, number of password resets, and total number of requests. 
     
     
         41 . The apparatus of  claim 31 , wherein the one or more statistical models comprises at least one of: Clustering models, Hidden Markov model, and Copula models. 
     
     
         42 . The apparatus of  claim 31 , wherein the one or more rules for identifying threats to the enterprise or e-commerce system comprises a random forest classifier, learning vector quantization, and/or a neural network. 
     
     
         43 . The apparatus of  claim 31 , the using one or more statistical models on the one or more features comprises using a bag of little bootstraps sampling. 
     
     
         44 . The apparatus of  claim 31 , wherein the using one or more statistical models on the one or more features tables to identify statistical outliers comprises:
 distributing one or more features from the one or more features tables across two or more servers;   using the one or more statistical models on the distributed one or more features; and   aggregating results from the using the one or more statistical models on the distributed one or more features.   
     
     
         45 . The apparatus of  claim 44 , wherein the using the one or more statistical models on the distributed one or more features comprises using a bag of little bootstraps sampling. 
     
     
         46 . The apparatus of  claim 31 , wherein the labeling the statistical outliers to create one or more labeled features tables comprises presenting an administrator the statistical outliers for identification as malicious, non-malicious, or other administrator defined label.