Updating search engine document index based on calculated age of changed portions in a document

Abstract

A system receives a document that includes new content and aged content, and compares the document with a prior version of the document that includes the aged content but not the new content. The system also separates the new content and the aged content based on the comparison, determines ages associated with the new content and the aged content, and determines whether the ages of the new content and the aged content are greater than or equal to an age threshold. The system further calculates a checksum of the document based on the aged content when the age of the aged content is greater than or equal to the age threshold, and the age of the new content is less than the age threshold, and stores the calculated checksum.

1. A computing device-implemented method, comprising:
 receiving a document; 
 comparing the document with a prior version of the document; 
 determining, based on the comparison, at least one first portion of the document that is common with the prior version of the document and at least one second portion of the document that is uncommon with the prior version of the document; 
 determining an age of the at least one first portion and the at least one second portion; 
 determining whether the age of the at least one first portion is greater than or equal to an age threshold and that the age of the at least one second portion is less than the age threshold; 
 calculating a checksum of the document based on the at least one first portion but not the at least one second portion when the age of the at least one first portion is greater than or equal to the age threshold and the age of the at least one second portion is less than the age threshold; and 
 storing the calculated checksum. 
 
     
     
       2. The method of  claim 1 , where comparing the document with a prior version of the document comprises one of:
 comparing the document with the prior version of the document using a longest common subsequence process; or 
 comparing the document with the prior version of the document using a winnowing process. 
 
     
     
       3. The method of  claim 1 , where determining the age of the at least one first portion and the at least one second portion comprises:
 analyzing an age parameter of a token associated with the at least one first portion; 
 analyzing an age parameter of a token associated with the at least on second portion; 
 increasing the age parameters each time the document is crawled; and 
 determining the age of the least one first portion and the at least one second portion based on the increased age parameters. 
 
     
     
       4. The method of  claim 1 , further comprising:
 determining whether the calculated checksum matches a checksum associated with a cluster of duplicate documents; and 
 associating the document with the cluster of duplicate documents when the calculated checksum matches the checksum associated with the cluster of duplicate documents. 
 
     
     
       5. The method of  claim 4 , further comprising:
 selecting one of the documents in the cluster of duplicate documents as representative of the cluster of duplicate documents; and 
 indexing the selected document. 
 
     
     
       6. The method of  claim 5 , where selecting one of the documents in the cluster of duplicate documents as representative of the cluster of duplicate documents comprises:
 creating a ranked list of the documents in the cluster of duplicate documents. 
 
     
     
       7. The method of  claim 4 , where, when the calculated checksum does not match the checksum associated with the cluster of duplicate documents, the method further comprises:
 creating a new cluster of duplicate documents; and 
 associating the document with the new cluster of duplicate documents. 
 
     
     
       8. The method of  claim 1 , where the at least one second portion comprises changing content. 
     
     
       9. The method of  claim 1 , where the at least one second portion comprises new content. 
     
     
       10. A computing device-implemented system, comprising:
 means for receiving an aged version of a document; 
 means for comparing the aged version of the document with a prior version of the document; 
 means for determining at least one first portion of the aged version of the document that is common with the prior version of the document based on the comparison; 
 means for determining at least one second portion of the aged version of the document that is uncommon with the prior version of the document based on the comparison; 
 means for determining an age of the at least one first portion and the at least one second portion; 
 means for determining whether the age of the at least one first portion is greater than or equal to an age threshold and that the age of the at least one second portion is less than the age threshold; 
 means for calculating a checksum of the aged version of the document based on the at least one first portion but not the at least one second portion when the age of the at least one first portion is greater than or equal to the age threshold and the age of the at least one second portion is less than the age threshold; 
 means for determining whether the calculated checksum matches a checksum associated with a cluster of duplicate documents; and 
 means for associating the aged version of the document with the cluster of duplicate documents when the calculated checksum matches the checksum associated with the cluster of duplicate documents. 
 
     
     
       11. The system of  claim 10 , further comprising:
 means for creating a new cluster of duplicate documents when the calculated checksum does not match the checksum associated with the cluster of duplicate documents; and 
 means for associating the aged version of the document with the new cluster of duplicate documents when the calculated checksum does not match the checksum associated with the cluster of duplicate documents. 
 
     
     
       12. A system, comprising:
 a memory to store a plurality of instructions; and 
 a processor to execute instructions in the memory to:
 receive a document, 
 compare the document with a prior version of the document, 
 determine, based on the comparison, at least one first portion of the document that is common with the prior version of the document, and at least one second portion of the document that is uncommon with the prior version of the document, 
 determine an age of the at least one first portion and the at least one second portion, 
 determine whether the age of the at least one first portion is greater than or equal to an age threshold and that the age of the at least one second portion is less than the age threshold, 
 calculate a checksum of the document based on the at least one first portion but not the at least one second portion when the age of the at least one first portion is greater than or equal to the age threshold and the age of the at least one second portion is less than the age threshold, and 
 associate the document with a cluster of duplicate documents when the calculated checksum matches the checksum associated with the cluster of duplicate documents. 
 
 
     
     
       13. The system of  claim 12 , where, when determining the age of the at least one first portion and the at least one second portion, the processor is configured to:
 analyze an age parameter of a token associated with the at least one first portion, 
 analyze an age parameter of a token associated with the at least on second portion, 
 increase the age parameters each time the document is crawled, and 
 determine the age of the least one first portion and the at least one second portion based on the increased age parameters. 
 
     
     
       14. The system of  claim 12 , where the processor is further configured to:
 create a new cluster of duplicate documents when the calculated checksum does not match the checksum associated with the cluster of duplicate documents, and 
 associate the document with the new cluster of duplicate documents when the calculated checksum does not match the checksum associated with the cluster of duplicate documents. 
 
     
     
       15. The system of  claim 14 , where the at least one second portion comprises one of changing content or new content. 
     
     
       16. The system of  claim 12 , where the processor is further configured to:
 select one of the documents in the cluster of duplicate documents as representative of the cluster of duplicate documents, and 
 index the selected document. 
 
     
     
       17. The system of  claim 12 , where the document includes new content and aged content, and when determining the at least one first portion and the at least one second portion, the processor is configured to:
 determine whether the at least one first portion comprises the aged content of the document, and 
 calculate the checksum of the document based solely on the aged content of the document. 
 
     
     
       18. The system of  claim 12 , where:
 the document comprises a currently-crawled version of the document, and 
 the prior version of the document comprises a previously-crawled version of the document. 
 
     
     
       19. A computing device-implemented method, comprising:
 receiving a document that includes new content and aged content; 
 comparing the document with a prior version of the document that includes the aged content but not the new content; 
 separating the new content and the aged content based on the comparison; 
 determining ages associated with the new content and the aged content; 
 determining whether the ages of the new content and the aged content are greater than or equal to an age threshold; 
 calculating a checksum of the document based on the aged content but not the new content when the age of the aged content is greater than or equal to the age threshold, and the age of the new content is less than the age threshold; and 
 storing the calculated checksum. 
 
     
     
       20. The method of  claim 19 , where determining ages associated with the new content and the aged content comprises:
 analyzing age parameters associated with the new content and the aged content; 
 increasing the age parameters each time the document is crawled; and 
 determining the ages of the new content and the aged content based on the increased age parameters. 
 
     
     
       21. The method of  claim 19 , further comprising:
 determining whether the calculated checksum matches a checksum associated with a cluster of duplicate documents; and 
 associating the document with the cluster of duplicate documents when the calculated checksum matches the checksum associated with the cluster of duplicate documents. 
 
     
     
       22. The method of  claim 21 , further comprising:
 selecting one of the documents in the cluster of duplicate documents as representative of the cluster of duplicate documents; and 
 indexing the selected document. 
 
     
     
       23. The method of  claim 21 , where, when the calculated checksum does not match the checksum associated with the cluster of duplicate documents, the method further comprises:
 creating a new cluster of duplicate documents; and 
 associating the document with the new cluster of duplicate documents. 
 
     
     
       24. One or more devices, comprising:
 a memory to store a plurality of instructions; and 
 a processor to execute instructions in the memory to:
 receive a document that includes new content and aged content, 
 compare the document with a prior version of the document, 
 determine, based on the comparison, the new content that is not in the prior version of the document and the aged content that is in the prior version of the document, 
 determine ages associated with the new content and the aged content, 
 determine whether the ages of the new content and the aged content are greater than or equal to an age threshold, 
 index the new content when the age of the new content is determined to be less than the age threshold, 
 determine, when the age of aged content is determined to be greater than or equal to the age threshold, whether the aged content is associated with a cluster of duplicate documents, and 
 associate the document with the cluster of duplicate documents when the aged content is associated with the cluster of duplicate documents. 
 
 
     
     
       25. A computing device-implemented method, comprising:
 receiving, via the computing device, a document that includes new content and aged content; 
 comparing, via a processor associated with the computing device, the document with a prior version of the document; 
 determining, via the processor and based on the comparison, the new content that is not in the prior version of the document and the aged content that is in the prior version of the document; 
 determining, via the processor, ages associated with the new content and the aged content; 
 determining, via the processor, whether the ages of the new content and the aged content are greater than or equal to an age threshold; 
 indexing, via the processor, the new content when the age of the new content is determined to be less than the age threshold; 
 storing the indexed new content in a memory associated with the computing device; 
 determining, via the processor and when the age of aged content is determined to be greater than or equal to the age threshold, whether the aged content is associated with a cluster of duplicate documents, and 
 associating, via the processor, the document with the cluster of duplicate documents when the aged content is associated with the cluster of duplicate documents.