System and method for document collection, grouping and summarization

Abstract

Computer-based method of generating a summary of one or more documents comprises identifying content including text having a measurable quality from a predetermined location, evaluating the content, using a computer processor, to determine whether the content represents a document of interest, and preparing a summary of the content if the content represents document of interest. A computer-based method of generating a summary of one or more documents, each including two or more sentences, is also provided.

1 . A computer-based method of generating a summary of one or more documents comprising:
 identifying content including text having a measurable quality from a predetermined location;   evaluating the content, using a computer processor, to determine whether the content represents a document of interest; and   preparing a summary of the content if the content represents document of interest.   
     
     
         2 . The computer-based method of  claim 1 , wherein the evaluating the content comprises extracting the text of the content and comparing the quantity of text to a predetermined threshold to determine whether the content represents a document of interest. 
     
     
         3 . The computer-based method of  claim 1 , wherein the identifying content from a predetermined location is performed by a document collection engine. 
     
     
         4 . The computer-based method of  claim 1 , wherein the predetermined location is a list of internet content provider websites. 
     
     
         5 . The computer-based method of  claim 4 , wherein a script or text filed is used to provide the list of internet content provider websites. 
     
     
         6 . The computer-based method of  claim 5 , wherein the list of internet sites is provided in the form of uniform resource locators (URL) for the sites. 
     
     
         7 . The computer-based method of  claim 1 , further comprising pre-processing the identified content to convert the identified content to a XML format. 
     
     
         8 . The computer-based method of  claim 2 , wherein the predetermined threshold is about 512 characters. 
     
     
         9 . The computer-based method of  claim 1 , further comprising determining if there are images associated with the content if the content represents a document of interest. 
     
     
         10 . The computer-based method of  claim 9 , wherein the determining if there are images associated with the content comprises determining if there are image files located within a same cell as the content. 
     
     
         11 . The computer-based method of  claim 10 , wherein the determining if there are images associated with the content further comprises determining whether the computer readable file format of the image file is representative of an image associated with the content. 
     
     
         12 . The computer-based method of  claim 11 , wherein the determining if there are images associated with the content further comprises rejecting image files having an address of the image file that is indicative of advertising content. 
     
     
         13 . A computer-based method of generating a summary of one or more documents, each including two or more sentences, comprising:
 assigning an importance score, using a computer processor, to each of the two or more sentences in the one or more documents;   ranking the sentences based upon the corresponding assigned importance score;   preparing a summary of the one or more documents by selecting the sentences in the order of the ranking until a predetermined summary length is attained;   reordering the sentences to improve readability of the summary;   and removing duplicate sentences from the summary.   
     
     
         14 . The computer-based method of  claim 13 , wherein the assigning an importance score comprises assigning a higher score if the sentence includes words associated with lead sentences of articles. 
     
     
         15 . The computer-based method of  claim 13 , wherein the assigning an importance score comprises assigning a higher score if the sentence includes highly specific verbs. 
     
     
         16 . The computer-based method of  claim 13 , wherein the assigning an importance score comprises building concept sets for each sentence by generalizing noun-verb pairs into classes of words which refer to one another. 
     
     
         17 . The computer-based method of  claim 13 , wherein the assigning an importance score comprises assigning a score based upon location of the sentence within the document. 
     
     
         18 . The computer-based method of  claim 17 , wherein the sentences are assigned a higher score if located near the beginning of the document. 
     
     
         19 . The computer-based method of  claim 13 , wherein the assigning an importance score comprises assigning a score based the length of the sentence. 
     
     
         20 . The computer-based method of  claim 19 , wherein the sentences are assigned a lower score if the length is less than a predetermined word count. 
     
     
         21 . The computer-based method of  claim 20 , wherein the predetermined word count is 15 words. 
     
     
         22 . The computer-based method of  claim 19 , wherein the sentences are assigned a lower score if the length is greater than a predetermined word count. 
     
     
         23 . The computer-based method of  claim 22 , wherein the predetermined word count is 30 words. 
     
     
         24 . The computer-based method of  claim 13 , wherein the assigning an importance score comprises assigning a score based on the frequency of occurrence of a named entity occurring in the document. 
     
     
         25 . The computer-based method of  claim 13 , wherein the assigning an importance score comprises assigning a lower score if the sentence begins with a pronoun. 
     
     
         26 . The computer-based method of  claim 13 , wherein the assigning an importance score comprises assigning a higher score if the sentence has a pronoun following a named entity. 
     
     
         27 . The computer-based method of  claim 13 , further comprising performing named entity substitution to improve the readability of the summary. 
     
     
         28 . The computer-based method of  claim 27 , wherein the named entity substitution includes replacing a first reference to a named entity in the summary with a longest variant of the named entity in the summary. 
     
     
         29 . The computer-based method of  claim 28 , wherein each subsequent reference to a named entity is replaced with a shortest variant of the named entity in the summary.