Facet Blacklisting in Anomaly Detection

Abstract

A security server receives a full hash and a set of subhashes from a client. The security server determines that the full hash is blacklisted. The security server updates, for each subhash in the set of subhashes, an associated malicious count. The security server adds a subhash to a subhash blacklist responsive to an associated malicious count exceeding a threshold. The security server receives a second set of subhashes. The security server determines whether at least one of the subhashes in the second set of subhashes is included in the subhash blacklist. The security server reports to the client based on the determination.

1 . A method comprising:
 receiving a first full hash and a first plurality of subhashes from a client, wherein the first full hash is a hash of a first file and each subhash in the first plurality of subhashes is a hash of a facet of the first file;   determining whether the first full hash is blacklisted;   responsive to determining the first full hash is blacklisted, updating, for each subhash in the first plurality of subhashes, an associated malicious count, wherein the malicious count tracks a historic number of blacklisted files with which the subhash is associated;   responsive to a first malicious count of the malicious counts exceeding a threshold malicious count, adding the subhash associated with the first malicious count to a subhash blacklist;   receiving a second full hash and a second plurality of subhashes from the client, wherein the second full hash is a hash of a second file and each subhash in the second plurality of subhashes is a hash of a facet of the second file;   determining whether the second full hash is blacklisted;   responsive to determining the second full hash is not blacklisted, determining whether a subhash in the second plurality of subhashes is included in the subhash blacklist;   responsive to determining a subhash in the second plurality of subhashes is included in the subhash blacklist, determining the second file is blacklisted; and   reporting that the second file is blacklisted to the client.   
     
     
         2 . The method of  claim 1 , further comprising:
 receiving a third plurality of subhashes from the client, wherein each subhash in the third plurality of subhashes is a hash of a facet of a third file;   determining that the third file is malicious; and   removing a subhash in the third plurality of subhashes from a subhash whitelist.   
     
     
         3 . The method of  claim 2 , wherein removing the subhash in the third plurality of subhashes from the subhash whitelist is responsive to a malicious count associated with the subhash in the third plurality of subhashes comprising a nonzero value. 
     
     
         4 . The method of  claim 1 , further comprising:
 receiving a third plurality of subhashes of facets of a third file from a second client;   determining whether at least one subhash in the third plurality of subhashes is included in the subhash blacklist, comprising:
 determining whether a subhash in the third plurality of subhashes is the subhash associated with the first malicious count; and 
   reporting a result of determining whether at least one subhash in the third plurality of subhashes is included in the subhash blacklist to the second client.   
     
     
         5 . The method of  claim 1 , further comprising:
 receiving a third plurality of subhashes from the client, wherein each subhash in the third plurality of subhashes is a hash of a facet of a third file;   determining whether at least one subhash in the third plurality of subhashes is included in a subhash whitelist; and   responsive to determining at least one subhash included in the third plurality of subhashes is included in the subhash whitelist, reporting that the third file is clean to the client.   
     
     
         6 . The method of  claim 1 , wherein reporting that the second file is blacklisted to the client comprises reporting that the second file is blacklisted to a protection application at the client. 
     
     
         7 . The method of  claim 1 , wherein the threshold malicious count is one. 
     
     
         8 . The method of  claim 1 , wherein reporting that the second file is blacklisted to the client comprises sending instructions to the client to perform at least one of removing the second file, quarantining the second file, and providing a notification to a user of the client. 
     
     
         9 . A non-transitory computer-readable storage medium storing computer program instructions executable by a processor to perform operations comprising:
 receiving a first full hash and a first plurality of subhashes from a client, wherein the first full hash is a hash of a first file and each subhash in the first plurality of subhashes is a hash of a facet of the first file;   determining whether the first full hash is blacklisted;   responsive to determining the first full hash is blacklisted, updating, for each subhash in the first plurality of subhashes, an associated malicious count, wherein the malicious count tracks a historic number of blacklisted files with which the subhash is associated;   responsive to a first malicious count of the malicious counts exceeding a threshold malicious count, adding the subhash associated with the first malicious count to a subhash blacklist;   receiving a second full hash and a second plurality of subhashes from the client, wherein the second full hash is a hash of a second file and each subhash in the second plurality of subhashes is a hash of a facet of the second file;   determining whether the second full hash is blacklisted;   responsive to determining the second full hash is not blacklisted, determining whether a subhash in the second plurality of subhashes is included in the subhash blacklist;   responsive to determining a subhash in the second plurality of subhashes is included in the subhash blacklist, determining the second file is blacklisted; and   reporting that the second file is blacklisted to the client.   
     
     
         10 . The non-transitory computer-readable storage medium of  claim 9 , the operations further comprising:
 receiving a third plurality of subhashes from the client, wherein each subhash in the third plurality of subhashes is a hash of a facet of a third file;   determining that the third file is malicious; and   removing a subhash in the third plurality of subhashes from a subhash whitelist.   
     
     
         11 . The non-transitory computer-readable storage medium of  claim 10 , wherein removing the subhash in the third plurality of subhashes from the subhash whitelist is responsive to a malicious count associated with the subhash in the third plurality of subhashes comprising a nonzero value. 
     
     
         12 . The non-transitory computer-readable storage medium of  claim 9 , the operations further comprising:
 receiving a third plurality of subhashes of facets of a third file from a second client;   determining whether at least one subhash in the third plurality of subhashes is included in the subhash blacklist, comprising:
 determining whether a subhash in the third plurality of subhashes is the subhash associated with the first malicious count; and 
   reporting a result of determining whether at least one subhash in the third plurality of subhashes is included in the subhash blacklist to the second client.   
     
     
         13 . The non-transitory computer-readable storage medium of  claim 9 , the operations further comprising:
 receiving a third plurality of subhashes from the client, wherein each subhash in the third plurality of subhashes is a hash of a facet of a third file;   determining whether at least one subhash in the third plurality of subhashes is included in a subhash whitelist; and   responsive to determining at least one subhash included in the third plurality of subhashes is included in the subhash whitelist, reporting that the third file is clean to the client.   
     
     
         14 . The non-transitory computer-readable storage medium of  claim 9 , wherein reporting that the second file is blacklisted to the client comprises reporting that the second file is blacklisted to a protection application at the client. 
     
     
         15 . The non-transitory computer-readable storage medium of  claim 9 , wherein the threshold malicious count is one. 
     
     
         16 . A method, comprising:
 receiving a full hash and a plurality of subhashes, wherein the full hash is a hash of a file and each subhash in the plurality of subhashes is a hash of a facet of the file;   determining whether the full hash is whitelisted;   responsive to determining the full hash is not whitelisted, determining whether a subhash in the plurality of subhashes is included in a subhash whitelist, wherein a subhash is included in the subhash whitelist responsive to a clean count associated with the subhash exceeding a threshold value, the clean count indicating a number of previous instances of a respective full hash being whitelisted;   responsive to determining a subhash in the plurality of subhashes is included in the subhash whitelist, determining the file is whitelisted; and   reporting that the file is whitelisted.   
     
     
         17 . The method of  claim 16 , wherein at least one facet is an author string, a product name string, a list of application programming interfaces, a file description, copyright information, or a portion of a file header. 
     
     
         18 . The method of  claim 16 , further comprising:
 receiving a plurality of files; and   applying the plurality of files to a file filter to produce a subset of files comprising a subset of the plurality of files to be checked against a subhash whitelist;   wherein the first file is a file in the subset of files.   
     
     
         19 . The method of  claim 18 , wherein the file filter produces the subset of files by performing steps comprising:
 generating, for each file in the plurality of files, an anomaly score;   evaluating each anomaly score against an anomaly score threshold; and   producing the subset of files by adding files of the plurality of files to the subset of files with anomaly scores exceeding the anomaly score threshold.   
     
     
         20 . The method of  claim 16 , further comprising:
 receiving the first file;   analyzing the first file to generate a plurality of facets;   hashing the first file to produce the first full hash; and   hashing each facet in the plurality of facets to generate the first plurality of subhashes.