Computer-Implemented System And Method For Identifying Similar Documents

Abstract

A computer-implemented system and method for identifying similar documents is provided. A set of documents is obtained. Each document in the set is divided into segments and the segments are hashed. The hashed segments of at least two of the documents are compared. Hashed segments shared between the at least two documents are identified. A number of the hashed segments shared between the at least two documents is summed and a total number of hashed segments within the at least two documents is summed. A ratio of similarity between the at least two documents is determined based on the number of shared hashed segments and the total number of hashed segments.

What is claimed is: 
     
         1 . A computer-implemented system for identifying similar documents, comprising:
 a set of documents; and   a server comprising memory and a central processing unit, wherein the central processing unit executes the following:
 a segment module to divide each document in the set into segments and to hash each of the segments; 
 a comparison module to compare the hashed segments of at least two of the documents; 
 an identification module to identify hashed segments shared between the at least two documents; 
 a sum module to sum a number of the hashed segments shared between the at least two documents and to sum a total number of hashed segments within the at least two documents; and 
 a determination module to determine a ratio of similarity between the at least two documents based on the number of shared hashed segments and the total number of hashed segments. 
   
     
     
         2 . A system according to  claim 1 , wherein the documents are each divided into overlapping segments. 
     
     
         3 . A system according to  claim 2 , further comprising:
 an order module to order the overlapping segments and the hashes of the segments by occurrence within each of the at least two documents for comparison.   
     
     
         4 . A system according to  claim 1 , wherein each segment comprises a length of N words. 
     
     
         5 . A system according to  claim 1 , further comprising:
 a near duplicate identification module to identify the at least two documents as near duplicate documents based on the ratio of similarity.   
     
     
         6 . A system according to  claim 1 , further comprising:
 a threshold module to apply a threshold to the similarity ratio and to designate as near duplicates the at least two documents having the similarity threshold that satisfies the threshold.   
     
     
         7 . A system according to  claim 6 , wherein the threshold is 0.2. 
     
     
         8 . A system according to  claim 1 , further comprising:
 a near duplicate identification module to identify the at least two documents as near duplicates, wherein one of the documents comprises spelling errors.   
     
     
         9 . A system according to  claim 1 , further comprising:
 a duplicate identification module to identify the at least two documents as duplicates based on the similarity ratio.   
     
     
         10 . A system according to  claim 1 , wherein the duplicate documents have a similarity value of 1.0. 
     
     
         11 . A computer-implemented method for identifying similar documents, comprising:
 obtaining a set of documents;   dividing each document in the set into segments and hashing each of the segments;   comparing the hashed segments of at least two of the documents;   identifying hashed segments shared between the at least two documents;   summing a number of the hashed segments shared between the at least two documents and summing a total number of hashed segments within the at least two documents; and   determining a ratio of similarity between the at least two documents based on the number of shared hashed segments and the total number of hashed segments.   
     
     
         12 . A method according to  claim 11 , wherein the documents are each divided into overlapping segments. 
     
     
         13 . A method according to  claim 12 , further comprising:
 ordering the overlapping segments and the hashes of the segments by occurrence within each of the at least two documents for comparison.   
     
     
         14 . A method according to  claim 11 , wherein each segment comprises a length of N words. 
     
     
         15 . A method according to  claim 11 , further comprising:
 identifying the at least two documents as near duplicate documents based on the ratio of similarity.   
     
     
         16 . A method according to  claim 11 , further comprising:
 applying a threshold to the similarity ratio of the at least two documents; and   designating as near duplicates the at least two documents sharing the similarity ratio that satisfies the threshold.   
     
     
         17 . A method according to  claim 16 , wherein the threshold is 0.2. 
     
     
         18 . A method according to  claim 11 , further comprising:
 identifying the at least two documents as near duplicates, wherein one of the documents comprises spelling errors.   
     
     
         19 . A method according to  claim 11 , further comprising:
 Identifying the at least two documents as duplicates based on the similarity ratio.   
     
     
         20 . A method according to  claim 19 , wherein the duplicate documents have a similarity value of 1.0.