Computer-Implemented System And Method For Identifying Relevant Documents

Abstract

A computer-implemented system and method for identifying relevant documents is provided. A set of documents each associated with one or more concepts is identified. At least a portion of the documents in the set are clustered based on the concepts. A matrix that provides a summary of the documents most relevant to one such concept is generated by determining for each document a measure of similarity between a concept frequency occurrence and concept weights of each cluster. The matrix is populated with the calculated measures of similarity. Those documents associated with a threshold measure of similarity are identified as the most relevant documents to one such concept.

What is claimed is: 
     
         1 . A computer-implemented system for identifying relevant documents, comprising:
 a database to store a set of documents each associated with one or more concepts; and   a server comprising a central processing unit, memory, an input port to receive the document set, and an output port, wherein the central processing unit is configured to:
 cluster at least a portion of the documents in the set based on the concepts; and 
 generate a matrix that provides a summary of those documents that are most relevant to one such concept by determining for each document a measure of similarity between a concept frequency occurrence and concept weights of each cluster and populating the matrix with the calculated measures of similarity; and 
 identify those documents associated with a threshold measure of similarity as the most relevant documents to one such concept. 
   
     
     
         2 . A system according to  claim 1 , wherein the central processing unit removes those documents from the matrix with zero values for one or more concept frequency occurrences. 
     
     
         3 . A system according to  claim 1 , wherein the central processing unit calculates the similarity according to the following equation: 
       
         
           
             
               
                 d 
                 cluster 
               
               = 
               
                 
                   ∑ 
                   
                     i 
                     → 
                     n 
                   
                 
                  
                 
                   
                     doc 
                     
                       term 
                       i 
                     
                   
                   · 
                   
                     cluster 
                     
                       term 
                       i 
                     
                   
                 
               
             
           
         
       
       where doc term  represents the frequency of occurrence for a given term i in one such document and cluster term  represents the weight of a given cluster for a given term i. 
     
     
         4 . A system according to  claim 1 , wherein the central processing unit updates the concept weights of the clusters until the clusters settle. 
     
     
         5 . A system according to  claim 1 , wherein the central processing unit removes duplicate documents from the identified documents that satisfy the threshold similarity. 
     
     
         6 . A system according to  claim 1 , wherein the central processing unit identifies the documents for clustering by creating a histogram for each document that maps a relative frequency of occurrence for at least a portion of terms in that document, creating a corpus graph that maps the concepts for the documents in the set based on the terms, defining predefined thresholds for the corpus graph, and selecting those documents with terms that satisfy the thresholds for clustering. 
     
     
         7 . A system according to  claim 6 , wherein the central processing unit removes from the set, those documents with terms that fail to satisfy the threshold. 
     
     
         8 . A system according to  claim 6 , wherein the thresholds are set at 1% to 15% for shorter documents. 
     
     
         9 . A system according to  claim 6 , wherein the central processing unit sets tighter thresholds for larger documents. 
     
     
         10 . A system according to  claim 1 , wherein the central processing unit extracts terms from the documents in the set and generates a record for each term. 
     
     
         11 . A computer-implemented method for identifying relevant documents, comprising:
 identifying a set of documents each associated with one or more concepts;   clustering at least a portion of the documents in the set based on the concepts;   generating a matrix that provides a summary of those documents that are most relevant to one such concept, comprising:
 determining for each document a measure of similarity between a concept frequency occurrence and concept weights of each cluster; and 
 populating the matrix with the calculated measures of similarity; and 
   identifying those documents associated with a threshold measure of similarity as the most relevant documents to one such concept.   
     
     
         12 . A method according to  claim 11 , further comprising:
 removing those documents from the matrix with zero values for one or more concept frequency occurrences.   
     
     
         13 . A method according to  claim 11 , further comprising:
 calculating the similarity according to the following equation:   
       
         
           
             
               
                 d 
                 cluster 
               
               = 
               
                 
                   ∑ 
                   
                     i 
                     → 
                     n 
                   
                 
                  
                 
                   
                     doc 
                     
                       term 
                       i 
                     
                   
                   · 
                   
                     cluster 
                     
                       term 
                       i 
                     
                   
                 
               
             
           
         
       
       where doc term  represents the frequency of occurrence for a given term i in one such document and cluster term  represents the weight of a given cluster for a given term i. 
     
     
         14 . A method according to  claim 11 , further comprising:
 updating the concept weights of the clusters until the clusters settle.   
     
     
         15 . A method according to  claim 11 , further comprising:
 removing duplicate documents from the identified documents that satisfy the threshold similarity.   
     
     
         16 . A method according to  claim 11 , further comprising:
 identifying the documents for clustering, comprising:
 creating a histogram for each document that maps a relative frequency of occurrence for at least a portion of terms in that document; 
 creating a corpus graph that maps the concepts for the documents in the set based on the terms; 
 defining predefined thresholds for the corpus graph; 
 selecting those documents with terms that satisfy the thresholds for clustering. 
   
     
     
         17 . A method according to  claim 16 , further comprising:
 removing from the set, those documents with terms that fail to satisfy the threshold.   
     
     
         18 . A method according to  claim 16 , wherein the thresholds are set at 1% to 15% for shorter documents. 
     
     
         19 . A method according to  claim 16 , further comprising:
 setting tighter thresholds for larger documents.   
     
     
         20 . A method according to  claim 11 , further comprising:
 extracting terms from the documents in the set; and   generating a record for each term.