Computer-Implemented System And Method For Clustering Documents Based On Scored Concepts

Abstract

A computer-implemented system and method for clustering documents based on scored concepts is provided. A set of documents is obtained and concepts are extracted from the documents. A score is calculated for each concept. The score is determined as a function of summation of a frequency of occurrence, concept weight, structural weight, and corpus weight. The documents in the set are clustered based on the scores. A vector is formed for each document based on the concepts in that document and the scores associated with the concepts. A similarity is determined between each document and each of the other documents based on the formed vectors. Those documents that are sufficiently distinct from the other documents are identified as seed documents for separate document clusters. Each of the remaining documents are grouped into one of the clusters most similar to that remaining document.

What is claimed is: 
     
         1 . A computer-implemented system for clustering documents based on scored concepts, comprising:
 a set of documents;   an extractor module to extract concepts from the document set;   a score module to calculate a score for each concept, comprising:
 a weight module to determine a frequency of occurrence of the concept within the document, a concept weight, a structural weight, and a corpus weight; and 
 a calculation module to determine the score for the concept as a function of summation of the frequency of occurrence, concept weight, structural weight, and corpus weight; and 
   a cluster module to cluster the documents in the set based on the scores, comprising:
 a vector module to form for each document, a vector based on the concepts in that document and the scores associated with the concepts; 
 a similarity module to determine a similarity between each document and each of the other documents based on the vectors; 
 a seed module to identify based on the similarities, those documents that are sufficiently distinct from the other documents as seed documents for separate document clusters; and 
 a grouping module to group each of the remaining documents into one of the document clusters most similar to that remaining document. 
   
     
     
         2 . A system according to  claim 1 , further comprising:
 a similarity determination module to determine for each cluster, a similarity of each document in that cluster to a center of the cluster.   
     
     
         3 . A system according to  claim 2 , further comprising:
 a deviation determination module to determine a standard deviation for the documents in the cluster based on the similarities to the cluster center;   a threshold module to apply a dynamic threshold to the standard deviation; and   an outlier module to identify those documents with similarities outside the dynamic threshold as outlier documents.   
     
     
         4 . A system according to  claim 3 , further comprising:
 an outlier processing module to process the outlier documents by determining a similarity between each outlier document and each of the cluster centers, determining a best fit between one such outlier document and one of the cluster centers, and grouping the outlier document into the cluster associated with the best fit cluster center.   
     
     
         5 . A system according to  claim 1 , further comprising:
 a display to display the scored concepts on a graph based on the scores.   
     
     
         6 . A system according to  claim 1 , wherein the concept weight provides a specificity of meaning for one such concept. 
     
     
         7 . A system according to  claim 1 , wherein the documents comprise at least one of structured and unstructured data. 
     
     
         8 . A system according to  claim 1 , further comprising:
 a database to store the documents; and   a document conversion module to convert each document into a document record prior to extracting the concepts.   
     
     
         9 . A system according to  claim 8 , further comprising:
 a parser to parse a structure of each document; and   a representation module to create a standardized representation of that document during the document conversion.   
     
     
         10 . A system according to  claim 8 , wherein the document record comprises at least one of an identifier for the corresponding document and contents of the document. 
     
     
         11 . A computer-implemented method for clustering documents based on scored concepts, comprising:
 obtaining a set of documents;   extracting concepts from the document set;   calculating a score for each concept, comprising:
 determining a frequency of occurrence of the concept within the document, a concept weight, a structural weight, and a corpus weight; and 
 determining the score for the concept as a function of summation of the frequency of occurrence, concept weight, structural weight, and corpus weight; and 
   clustering the documents in the set based on the scores, comprising:
 forming for each document, a vector for each document based on the concepts in that document and the scores associated with the concepts; 
 determining a similarity between each document and each of the other documents based on the formed vectors; 
 identifying those documents that are sufficiently distinct from the other documents as seed documents for separate document clusters; and 
 grouping each of the remaining documents into one of the clusters most similar to that remaining document. 
   
     
     
         12 . A method according to  claim 11 , further comprising:
 for each cluster, determining a similarity of each document in that cluster to a center of the cluster.   
     
     
         13 . A method according to  claim 12 , further comprising:
 determining a standard deviation of the documents in the cluster based on the similarities to the cluster center;   applying a dynamic threshold to the standard deviation; and   identifying those documents with similarities outside the standard deviation as outlier documents.   
     
     
         14 . A method according to  claim 13 , further comprising:
 processing the outlier documents, comprising:
 determining a similarity between each outlier document and each other cluster center; 
 determining a best fit between one such outlier document and one of the cluster centers; and 
 grouping the outlier document into the cluster associated with the best fit cluster center. 
   
     
     
         15 . A method according to  claim 11 , further comprising:
 displaying the scored concepts on a graph based on the scores.   
     
     
         16 . A method according to  claim 11 , wherein the concept weight provides a specificity of a meaning for one such concept. 
     
     
         17 . A method according to  claim 11 , wherein the documents comprise at least one of structured and unstructured data. 
     
     
         18 . A method according to  claim 11 , further comprising:
 retrieving the documents from a database; and   converting each document into a document record prior to extracting the concepts.   
     
     
         19 . A method according to  claim 18 , further comprising:
 parsing through a structure of each document; and   creating a standardized representation of the document during the conversion of the document.   
     
     
         20 . A method according to  claim 18 , wherein the document record comprises at least one of identification of the corresponding document and contents of the document.