Apparatus method and medium for detecting payload anomaly using n-gram distribution of normal data

Abstract

A method, apparatus and medium are provided for detecting anomalous payloads transmitted through a network. The system receives payloads within the network and determines a length for data contained in each payload. A statistical distribution is generated for data contained in each payload received within the network, and compared to a selected model distribution representative of normal payloads transmitted through the network. The model payload can be selected such that it has a predetermined length range that encompasses the length for data contained in the received payload. Anomalous payloads are then identified based on differences detected between the statistical distribution of received payloads and the model distribution. The system can also provide for automatic training and incremental updating of models.

1 - 67 . (canceled) 
     
     
         68 . A method for identifying malicious payloads, the method comprising:
 receiving, using a hardware processor, a file identified as corresponding to a first file type from a first source;   generating a statistical distribution of the data included in the file received from the first source;   retrieving a model statistical distribution representative of the first file type, wherein the model statistical distribution is associated with a tolerance;   comparing the statistical distribution of the data included in the file to the model statistical distribution;   determining, based on the comparison of the statistical distribution of the data included in the file to the model statistical distribution, whether statistical distribution of the data included in the file is outside of the tolerance; and   in response to determining that the statistical distribution of the data included in the file is outside of the tolerance, determining that the file is likely to include a malicious program.   
     
     
         69 . The method of  claim 68 , further comprising inhibiting the file from being executed based on the determination that the file is likely to include a malicious program. 
     
     
         70 . The method of  claim 68 , further comprising, in response to determining that the file is likely to include a malicious program, causing the file to be tested to determine whether the file includes a malicious program. 
     
     
         71 . The method of  claim 68 , wherein the statistical distribution of the file is a byte value statistical distribution, and the retrieved model distribution is a byte value statistical distribution of files of the first file type. 
     
     
         72 . The method of  claim 68 , further comprising comparing the statistical distribution of the file to a plurality of statistical distributions each corresponding to known malicious programs. 
     
     
         73 . The method of  claim 72 , further comprising weighting byte values in the statistical distribution of file that correspond to a machine executable code or a script file with a higher weight than other byte values. 
     
     
         74 . The method of  claim 72 , further comprising:
 determining that the file does not correspond to any of the statistical distributions corresponding to the known malicious programs based on the comparison of the statistical distribution of the file to the plurality of statistical distributions;   identifying information corresponding to the malicious program; and   generating a signature for the malicious program based on the information corresponding to the malicious program.   
     
     
         75 . A system for identifying malicious payloads, the system comprising:
 a hardware processor that is programmed to:
 receive a file identified as corresponding to a first file type from a first source; 
 generate a statistical distribution of the data included in the file received from the first source; 
 retrieve a model statistical distribution representative of the first file type; 
 compare the statistical distribution of the data included in the file to the model statistical distribution; 
 determine, based on the comparison of the statistical distribution of the data included in the file to the model statistical distribution, whether statistical distribution of the data included in the file is outside of the tolerance; and 
 in response to determining that the statistical distribution of the data included in the file is outside of the tolerance, determine that the file is likely to include a malicious program. 
   
     
     
         76 . The system of  claim 75 , wherein the hardware processor is further programmed to inhibit the file from being executed based on the determination that the file is likely to include a malicious program. 
     
     
         77 . The system of  claim 75 , wherein the hardware processor is further programmed to, in response to determining that the file is likely to include a malicious program, cause the file to be tested to determine whether the file includes a malicious program. 
     
     
         78 . The system of  claim 75 , wherein the statistical distribution of the file is a byte value statistical distribution, and the retrieved model distribution is a byte value statistical distribution of files of the first file type. 
     
     
         79 . The system of  claim 75 , wherein the hardware processor is further programmed to compare the statistical distribution of the file to a plurality of statistical distributions each corresponding to known malicious programs. 
     
     
         80 . The system of  claim 79 , wherein the hardware processor is further programmed to weight byte values in the statistical distribution of file that correspond to a machine executable code or a script file with a higher weight than other byte values. 
     
     
         81 . The system of  claim 79 , wherein the hardware processor is further programmed to:
 determine that the file does not correspond to any of the statistical distributions corresponding to the known malicious programs based on the comparison of the statistical distribution of the file to the plurality of statistical distributions;   identify information corresponding to the malicious program; and   generate a signature for the malicious program based on the information corresponding to the malicious program.   
     
     
         82 . A non-transitory computer-readable medium containing instructions that, when executed by a processor, cause the processor to perform method for identifying malicious payloads, the method comprising:
 receiving a file identified as corresponding to a first file type from a first source;   generating a statistical distribution of the data included in the file received from the first source;   retrieving a model statistical distribution representative of the first file type;   comparing the statistical distribution of the data included in the file to the model statistical distribution;   determining, based on the comparison of the statistical distribution of the data included in the file to the model statistical distribution, whether statistical distribution of the data included in the file is outside of the tolerance; and   in response to determining that the statistical distribution of the data included in the file is outside of the tolerance, determining that the file is likely to include a malicious program.   
     
     
         83 . The non-transitory computer-readable medium of  claim 82 , wherein the method further comprises inhibiting the file from being executed based on the determination that the file is likely to include a malicious program. 
     
     
         84 . The non-transitory computer-readable medium of  claim 82 , wherein the method further comprises, in response to determining that the file is likely to include a malicious program, causing the file to be tested to determine whether the file includes a malicious program. 
     
     
         85 . The non-transitory computer-readable medium of  claim 82 , wherein the statistical distribution of the file is a byte value statistical distribution, and the retrieved model distribution is a byte value statistical distribution of files of the first file type. 
     
     
         86 . The non-transitory computer-readable medium of  claim 82 , wherein the method further comprises comparing the statistical distribution of the file to a plurality of statistical distributions each corresponding to known malicious programs. 
     
     
         87 . The non-transitory computer-readable medium of  claim 86 , wherein the method further comprises weighting byte values in the statistical distribution of file that correspond to a machine executable code or a script file with a higher weight than other byte values. 
     
     
         88 . The non-transitory computer-readable medium of  claim 86 , wherein the method further comprises:
 determining that the file does not correspond to any of the statistical distributions corresponding to the known malicious programs based on the comparison of the statistical distribution of the file to the plurality of statistical distributions;   identifying information corresponding to the malicious program; and   generating a signature for the malicious program based on the information corresponding to the malicious program.