Computer-Implemented System and Method for Identifying Documents for Review

Abstract

A computer-implemented system and method for identifying documents for review is provided. A set of documents is sorted by topic. A status of near duplicate is assigned to two or more documents in the set. Threads of the documents are identified. Each thread includes at least one near duplicate document that is recursively included within another near duplicate document in that thread. In each thread, the near duplicate documents are ordered based on a length of the thread included in each near duplicate document. The near duplicate document that has the longest length is selected for each thread and provided for review.

What is claimed is: 
     
         1 . A computer-implemented system for identifying documents for review, comprising:
 a set of documents sorted by topic; and   a processor to execute modules, comprising:
 an assignment module to assign a status of near duplicate to two or more documents in the set; 
 an identification module to identify threads of the documents, each thread comprising two or more near duplicate documents, wherein at least one of the near duplicate documents is recursively included within another one of the near duplicate documents in that thread; 
 an order module to order the near duplicate documents in each thread based on length of the thread included in each near duplicate document; 
 a document selection module to select for each thread the near duplicate document that has the longest length; and 
 a delivery module to provide the selected near duplicate documents for review. 
   
     
     
         2 . A system according to  claim 1 , further comprising:
 a status identification module to identify those documents in the set that are assigned a status of unique and those documents that are assigned a status of duplicate.   
     
     
         3 . A system according to  claim 2 , further comprising:
 a document association module to identify for each unique document in the set at least one of the near duplicate documents and the duplicate documents of that unique document; and   an entry module to enter the unique documents in a log and to group for each entered unique document the identified near duplicate documents and duplicate documents for that unique document.   
     
     
         4 . A system according to  claim 3 , wherein the near duplicate, duplicate, and unique documents are logged by parsing one or more fields from each of the logged documents, storing as a document record in the log, the parsed fields with content of the logged document, and associating a hash code with each document record in the log. 
     
     
         5 . A system according to  claim 4 , wherein the hash codes are utilized to identify the near duplicate, duplicate, and unique documents. 
     
     
         6 . A system according to  claim 2 , wherein the delivery module provides the unique documents with the selected near duplicate documents for review. 
     
     
         7 . A system according to  claim 1 , further comprising:
 a removal module to remove the remaining near duplicate documents in the thread for review.   
     
     
         8 . A system according to  claim 1 , wherein the near duplicate documents with the longest thread length are each identified by identifying a number of thread markers for each of the near duplicate documents in that thread and determining the near duplicate document with a highest number of the thread markers as having the longest thread length. 
     
     
         9 . A system according to  claim 8 , wherein the markers comprise one or more of a separator and a subject line indicator. 
     
     
         10 . A system according to  claim 1 , further comprising:
 a counter to identify a maximum number of the documents for processing during a single run and to calculate a number of passes to process the documents in the set based on the maximum number of documents identified.   
     
     
         11 . A computer-implemented method for identifying documents for review, comprising:
 sorting a set of documents for review by topic;   assigning a status of near duplicate to two or more documents in the set;   identifying threads of the documents, each thread comprising at least one near duplicate document recursively included within another one of the near duplicate documents in that thread;   ordering the near duplicate documents in each thread based on length of the thread included in each near duplicate document;   selecting for each thread the near duplicate document that has the longest length; and   providing the selected near duplicate documents for review.   
     
     
         12 . A method according to  claim 11 , further comprising:
 identifying in the set those documents that are assigned a status of unique and those documents that are assigned a status of duplicate.   
     
     
         13 . A method according to  claim 12 , further comprising:
 for each unique document in the set identifying at least one of the near duplicate documents and the duplicates documents of that unique document;   entering in a log each unique document; and   grouping for each unique document in the log the identified near duplicate documents and duplicate documents for that unique document.   
     
     
         14 . A method according to  claim 13 , further comprising:
 logging the near duplicate, duplicate, and unique documents, comprising:
 parsing one or more fields from each of the logged documents; 
 storing each logged document as a document record in the log, the document record comprising the parsed fields with content of that logged document; and 
 associating a hash code to each document record in the log. 
   
     
     
         15 . A method according to  claim 14 , further comprising:
 utilizing the hash codes to identify the near duplicate, duplicate, and unique documents.   
     
     
         16 . A method according to  claim 12 , further comprising:
 providing the unique documents with the selected near duplicate documents for review.   
     
     
         17 . A method according to  claim 11 , further comprising:
 removing for review the remaining near duplicate documents in the thread.   
     
     
         18 . A method according to  claim 11 , further comprising:
 identifying the near duplicate documents with the longest thread length, comprising:
 for each thread identifying a number of thread markers for each of the near duplicate documents in that thread; and 
 determining for each thread the near duplicate document with a highest number of the thread markers as having the longest thread length for that thread. 
   
     
     
         19 . A method according to  claim 18 , wherein the markers comprise one or more of a separator and a subject line indicator. 
     
     
         20 . A method according to  claim 11 , further comprising:
 identifying a maximum number of the documents for processing during a single run; and   calculating a number of passes to process the documents in the set based on the maximum number of documents identified.