Faster web application scans of web page data based on deduplication

Abstract

In an embodiment, a web application scanner (WAS) component retrieves first web page data associated with a first web page, and produces first deduplicated web page data associated with the first web page data by performing multiple deduplication functions to remove specific elements from the first web page data. The WAS component selectively scans the first web page via a security scan function based on a degree of similarity between the first deduplicated web page data and second deduplicated web page data that is deduplicated from second web page data associated with a second web page.

1 . A method of operating a web application scanner component, comprising:
 retrieving a set of first web page vector elements associated with a first web page;   producing a set of first deduplicated web page vector elements associated with the set of first web page vector elements by performing (i) a first deduplication function on the set of first web page vector elements to remove one or more non-auditable vector elements from the set of first web page vector elements, and (ii) a second deduplication function on the set of first web page vector elements to remove one or more attack vector elements from the set of first web page vector elements that are already queued for scanning via a security scan function or have already been scanned via the security scan function; and   selectively scanning the first web page via the security scan function based on a degree of similarity between the set of first deduplicated web page vector elements and a set of second deduplicated web page vector elements that is deduplicated from a set of second web page vector elements associated with a second web page.   
     
     
         2 . The method of  claim 1 , wherein the set of second web page vector elements is queued for scanning via the security scan function or has already been scanned via the security scan function. 
     
     
         3 . The method of  claim 1 , wherein the set of first deduplicated web page vector elements comprises a hash of a portion of the set of first web page vector elements that excludes the one or more non-auditable vector elements and the one or more attack vector elements. 
     
     
         4 . The method of  claim 1 , wherein the one or more non-auditable vector elements that are removed from the set of first web page vector elements via the first deduplication function include:
 web page transition content, or   text content, or   image content, or   video content, or   audio content, or   style or formatting content, or   any combination thereof.   
     
     
         5 . The method of  claim 1 , wherein the one or more attack vector elements comprise:
 one or more forms, or   one or more cookies, or   any combination thereof.   
     
     
         6 . The method of  claim 1 , wherein the selectively scanning scans the first web page via one or more web page plugins if the degree of similarity between the set of first deduplicated web page vector elements and the set of second deduplicated web page vector elements exceeds a similarity threshold. 
     
     
         7 . The method of  claim 6 , wherein the similarity threshold is exceeded if the set of first deduplicated web page vector elements includes at least one attack vector element that is not part of the set of second deduplicated web page vector elements. 
     
     
         8 . The method of  claim 6 , wherein, if the degree of similarity between the set of first deduplicated web page vector elements and the set of second deduplicated web page vector elements does not exceed the similarity threshold, the selectively scanning comprises:
 scanning a server associated with the first web page via one or more server plugins of the security scan function,   scanning a uniform resource locator (URL) associated with the first web page via one or more URL plugins of the security scan function, and   skipping a scanning of the first web page via the one or more web page plugins of the security scan function.   
     
     
         9 . A web application scanner component, comprising:
 a memory; and   at least one processor communicatively coupled to the memory, the at least one processor configured to:   retrieve a set of first web page vector elements associated with a first web page;   produce a set of first deduplicated web page vector elements associated with the set of first web page vector elements by performing (i) a first deduplication function on the set of first web page vector elements to remove one or more non-auditable vector elements from the set of first web page vector elements, and (ii) a second deduplication function on the set of first web page vector elements to remove one or more attack vector elements from the set of first web page vector elements that are already queued for scanning via a security scan function or have already been scanned via the security scan function; and   selectively scan the first web page via the security scan function based on a degree of similarity between the set of first deduplicated web page vector elements and a set of second deduplicated web page vector elements that is deduplicated from a set of second web page vector elements associated with a second web page.   
     
     
         10 . The web application scanner component of  claim 9 , wherein the set of second web page vector elements is queued for scanning via the security scan function or has already been scanned via the security scan function. 
     
     
         11 . The web application scanner component of  claim 9 , wherein the set of first deduplicated web page vector elements comprises a hash of a portion of the set of first web page vector elements that excludes the one or more non-auditable vector elements and the one or more attack vector elements. 
     
     
         12 . The web application scanner component of  claim 9 , wherein the one or more non-auditable vector elements that are removed from the set of first web page vector elements via the first deduplication function include:
 web page transition content, or   text content, or   image content, or   video content, or   audio content, or   style or formatting content, or   any combination thereof.   
     
     
         13 . The web application scanner component of  claim 9 , wherein the one or more attack vector elements comprise:
 one or more forms, or   one or more cookies, or   any combination thereof.   
     
     
         14 . The web application scanner component of  claim 9 , wherein the selectively scanning scans the first web page via one or more web page plugins if the degree of similarity between the set of first deduplicated web page vector elements and the set of second deduplicated web page vector elements exceeds a similarity threshold. 
     
     
         15 . The web application scanner component of  claim 14 , wherein the similarity threshold is exceeded if the set of first deduplicated web page vector elements includes at least one attack vector element that is not part of the set of second deduplicated web page vector elements. 
     
     
         16 . The web application scanner component of  claim 14 , wherein, if the degree of similarity between the set of first deduplicated web page vector elements and the set of second deduplicated web page vector elements does not exceed the similarity threshold, the selectively scanning comprises:
 scan a server associated with the first web page via one or more server plugins of the security scan function,   scan a uniform resource locator (URL) associated with the first web page via one or more URL plugins of the security scan function, and   skip a scanning of the first web page via the one or more web page plugins of the security scan function.   
     
     
         17 . A non-transitory computer-readable medium storing computer-executable instructions that, when executed by a web application scanner component, cause the web application scanner component to:
 retrieve a set of first web page vector elements associated with a first web page;   produce a set of first deduplicated web page vector elements associated with the set of first web page vector elements by performing (i) a first deduplication function on the set of first web page vector elements to remove one or more non-auditable vector elements from the set of first web page vector elements, and (ii) a second deduplication function on the set of first web page vector elements to remove one or more attack vector elements from the set of first web page vector elements that are already queued for scanning via a security scan function or have already been scanned via the security scan function; and   selectively scan the first web page via the security scan function based on a degree of similarity between the set of first deduplicated web page vector elements and a set of second deduplicated web page vector elements that is deduplicated from a set of second web page vector elements associated with a second web page.   
     
     
         18 . The non-transitory computer-readable medium of  claim 17 , wherein the set of second web page vector elements is queued for scanning via the security scan function or has already been scanned via the security scan function. 
     
     
         19 . The non-transitory computer-readable medium of  claim 17 , wherein the set of first deduplicated web page vector elements comprises a hash of a portion of the set of first web page vector elements that excludes the one or more non-auditable vector elements and the one or more attack vector elements. 
     
     
         20 . The non-transitory computer-readable medium of  claim 17 , wherein the one or more non-auditable vector elements that are removed from the set of first web page vector elements via the first deduplication function include:
 web page transition content, or   text content, or   image content, or   video content, or   audio content, or   style or formatting content, or   any combination thereof.   
     
     
         21 . The non-transitory computer-readable medium of  claim 17 , wherein the one or more attack vector elements comprise:
 one or more forms, or   one or more cookies, or   any combination thereof.   
     
     
         22 . The non-transitory computer-readable medium of  claim 17 , wherein the selectively scanning scans the first web page via one or more web page plugins if the degree of similarity between the set of first deduplicated web page vector elements and the set of second deduplicated web page vector elements exceeds a similarity threshold. 
     
     
         23 . The non-transitory computer-readable medium of  claim 22 , wherein the similarity threshold is exceeded if the set of first deduplicated web page vector elements includes at least one attack vector element that is not part of the set of second deduplicated web page vector elements. 
     
     
         24 . The non-transitory computer-readable medium of  claim 22 , wherein, if the degree of similarity between the set of first deduplicated web page vector elements and the set of second deduplicated web page vector elements does not exceed the similarity threshold, the selectively scanning comprises:
 scan a server associated with the first web page via one or more server plugins of the security scan function,   scan a uniform resource locator (URL) associated with the first web page via one or more URL plugins of the security scan function, and   skip a scanning of the first web page via the one or more web page plugins of the security scan function.