URL normalization

Abstract

In one embodiment, a method includes receiving a plurality of uniform resource identifiers (URI's) associated with a particular domain. Each of the URI's identifies a content page comprising one or more signature elements. The method further includes, for each URI in the plurality of URI's, successively testing the URI to identify a core of the URI and any unnecessary elements of the URI. The core of the URI is sufficient to retrieve a version of the content page including all of its signature elements. The method additionally includes, for each URI in the plurality of URI's, updating a set of rules based on the identified core and the identified unnecessary elements. The set of rules establishes a normalized version of the URI.

What is claimed is: 
     
       1. A method comprising:
 by one or more processors, receiving a plurality of uniform resource identifiers (URI's) associated with a particular domain, wherein each of the URI's identifies a content page comprising one or more signature elements; and 
 by the one or more processors, for each URI in the plurality of URI's:
 successively retrieving content pages for modified versions of the URI to identify a core of the URI and any unnecessary elements of the URI, wherein the core of the URI corresponds to a modified version with the fewest number of URI elements that is sufficient to retrieve a version of the content page including all of its signature elements; and 
 updating a set of rules based on the identified core and the identified unnecessary elements, wherein the set of rules establishes a normalized version of the URI. 
 
 
     
     
       2. The method of  claim 1 , wherein at least one of the unnecessary elements of the URI is a parameter of the URI. 
     
     
       3. The method of  claim 1 , wherein at least one of the unnecessary elements of the URI is a section of a path of the URI. 
     
     
       4. The method of  claim 1 , wherein successively testing the URI comprises:
 determining a last element of the URI; 
 removing the last element from the URI to define an augmented URI; and 
 retrieving the identified content page for the augmented URI. 
 
     
     
       5. The method of  claim 1 , wherein successively testing the URI comprises:
 for each element of the URI and until the core of the URI is identified:
 determining a last element of the URI; 
 removing the last element of the URI to define an augmented URI; 
 retrieving the identified content page for the augmented URI; and 
 determining whether the retrieved content page includes all of the signature elements. 
 
 
     
     
       6. The method of  claim 5 , further comprising identifying the core of the URI as the augmented URI and the last element of the URI in response to determining that the retrieved content page does not include all of the signature elements. 
     
     
       7. The method of  claim 5 , wherein successively testing the URI further comprises:
 in response to determining that the identified content page for the augmented URI does not include all of the signature elements, determining that the last element of the URI is a key element of the core URI, the key element being necessary to retrieve the version of the content page including all of its signature elements. 
 
     
     
       8. The method of  claim 1 , wherein the URI comprises a path and a query string, the query string comprising one or more field-value pairs. 
     
     
       9. The method of  claim 8 , wherein successively testing the URI comprises successively removing field-value pairs from an end of the URI. 
     
     
       10. The method of  claim 1 , wherein the signature elements of the content page comprise one or more product listings. 
     
     
       11. The method of  claim 1 , wherein each normalized version of the URI is configured to retrieve a set of product listings. 
     
     
       12. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
 receive a plurality of uniform resource identifiers (URI's) associated with a particular domain, wherein each of the URI's identifies a content page comprising one or more signature elements; and 
 for each URI in the plurality of URI's:
 successively retrieving content pages for modified versions of the URI to identify a core of the URI and any unnecessary elements of the URI, wherein the core of the URI corresponds to a modified version with the fewest number of URI elements that is sufficient to retrieve a version of the content page including all of its signature elements; and 
 update a set of rules based on the identified core and the identified unnecessary elements, wherein the set of rules establishes a normalized version of the URI. 
 
 
     
     
       13. The computer-readable non-transitory storage media of  claim 12 , wherein at least one of the unnecessary elements of the URI is a parameter of the URI. 
     
     
       14. The computer-readable non-transitory storage media of  claim 12 , wherein at least one of the unnecessary elements of the URI is a section of a path of the URI. 
     
     
       15. The computer-readable non-transitory storage media of  claim 12 , wherein successively testing the URI comprises:
 determining a last element of the URI; 
 removing the last element from the URI to define an augmented URI; and 
 retrieving the identified content page for the augmented URI. 
 
     
     
       16. The computer-readable non-transitory storage media of  claim 12 , wherein successively testing the URI comprises:
 for each element of the URI and until the core of the URI is identified:
 determining a last element of the URI; 
 removing the last element of the URI to define an augmented URI; 
 retrieving the identified content page for the augmented URI; and 
 determining whether the retrieved content page includes all of the signature elements. 
 
 
     
     
       17. The computer-readable non-transitory storage media of  claim 16 , wherein the software is further operable when executed to identify the core of the URI as the augmented URI and the last element of the URI in response to determining that the retrieved content page does not include all of the signature elements. 
     
     
       18. The computer-readable non-transitory storage media of  claim 16 , wherein successively testing the URI further comprises:
 in response to determining that the identified content page for the augmented URI does not include all of the signature elements, determining that the last element of the URI is a key element of the core URI, the key element being necessary to retrieve the version of the content page including all of its signature elements. 
 
     
     
       19. The computer-readable non-transitory storage media of  claim 12 , wherein the URI comprises a path and a query string, the query string comprising one or more field-value pairs. 
     
     
       20. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
 receive a plurality of uniform resource identifiers (URI's) associated with a particular domain, wherein each of the URI's identifies a content page comprising one or more signature elements; and 
 for each URI in the plurality of URI's:
 successively retrieving content pages for modified versions of the URI to identify a core of the URI and any unnecessary elements of the URI, wherein the core of the URI corresponds to a modified version with the fewest number of URI elements that is sufficient to retrieve a version of the content page including all of its signature elements; and 
 update a set of rules based on the identified core and the identified unnecessary elements, wherein the set of rules establishes a normalized version of the URI.