Managed document research domains

Abstract

Techniques are disclosed for improving the quality and durability of citations used within research documents. A method and apparatus may be sued to enhance the ability of researches to collaborate (or to simply be alerted) when the substance of a reference cited in one research document changes in a way that may conflict with how that reference is used in the citing document. For example, changes in cited reference may undermine (or enhance) the conclusions of a research paper stored in the managed repository.

1 - 21 . (canceled) 
     
     
         22 . A computer-implemented method to monitor documents in a managed research domain to determine whether changes in cited content undermine semantic conformity with citing content, the computer-implemented method comprising:
 extracting one or more citations from a primary research document, wherein each extracted citation references a respective one of one or more cited documents;   for each cited document, evaluating content of the cited document using a set of one or more annotators to derive a respective collection of cited assertions from the content of the cited document;   evaluating content of the primary research document using the set of one or more annotators to derive a collection of citing assertions from the content of the primary research document;   subsequent to storing the primary research document, each collection of cited assertions, and the collection of citing assertions in a repository provided by the managed research domain, monitoring for changes to cited content in each cited document, including changes to cited content a first cited document;   for each monitored change to the cited content in the first cited document, determining, by operation of one or more computer processors and based on the derived collections of cited assertions and citing assertions, whether the respective monitored change undermines semantic conformity of the cited content in the first cited document with corresponding, citing content in the primary research document; and   upon determining that a first of the monitored changes to the cited content in the first cited document undermines semantic conformity with the citing content, designating the citing content as potentially lacking semantic conformity with the changed, cited content, wherein an indication of the citing content as being designated as potentially lacking semantic conformity is output.   
     
     
         23 . The computer-implemented method of  claim 22 , wherein at least one of the set of annotators normalizes terms in the primary research document and terms in the one or more cited documents to the common ontology. 
     
     
         24 . The computer-implemented method of  claim 22 , wherein determining that the first monitored change to the cited content undermines semantic conformity with the citing content comprises:
 comparing the assertions in the collection of cited assertions of the cited document containing the cited content, and the assertions in the collection of citing assertions to identify any significant conflicts between the collection of cited assertions of the cited document and the collection of citing assertions.   
     
     
         25 . The computer-implemented method of  claim 22 , further comprising scheduling each of the one or more cited references for a subsequent re-evaluation at a designated time. 
     
     
         26 . The computer-implemented method of  claim 22 , wherein a given one of the cited documents comprises a cited research paper not written by an author of the primary research paper, the computer-implemented method further comprising:
 at the designated time, for the given one of the cited documents, re-evaluating the cited reference using the set of annotators to derive an updated collection of cited assertions regarding the cited reference;   comparing the assertions in the collection of cited assertions and the assertions in the updated collection of cited assertions to identify any conflicts between the collection of cited assertions and the updated collection cited of assertions; and   in response to identifying a conflict and without requiring any user input, programmatically notifying the author of the primary research document that the cited research paper has changed.   
     
     
         27 . A non-transitory computer-readable medium to monitor documents in a managed research domain to determine whether changes in cited content undermine semantic conformity with citing content, the non-transitory computer-readable medium containing a program executable to perform an operation comprising:
 extracting one or more citations from a primary research document, wherein each extracted citation references a respective one of the one or more cited documents;   for each cited document, evaluating content of the cited document using a set of one or more annotators to derive a respective collection of cited assertions from the content of the cited document;   evaluating content of the primary research document using the set of one or more annotators to derive a collection of citing assertions from the content of the primary research document;   subsequent to storing the primary research document, each collection of cited assertions, and the collection of citing assertions in a repository provided by the managed research domain, monitoring for changes to cited content in each cited document, including changes to cited content a first cited document;   for each monitored change to the cited content in the first cited document, determining, by operation of one or more computer processors when executing the program, and based on the derived collections of cited assertions and citing assertions, whether the respective monitored change undermines semantic conformity of the cited content in the first cited document with corresponding, citing content in the primary research document; and   upon determining that a first of the monitored changes to the cited content in the first cited document undermines semantic conformity with the citing content, designating the citing content as potentially lacking semantic conformity with the changed, cited content, wherein an indication of the citing content as being designated as potentially lacking semantic conformity is output.   
     
     
         28 . The non-transitory computer-readable medium of  claim 27 , wherein at least one of the set of annotators normalizes terms in the primary research document and terms in the one or more cited documents to the common ontology. 
     
     
         29 . The non-transitory computer-readable medium of  claim 27 , wherein determining that the first monitored change to the cited content undermines semantic conformity with the citing content comprises:
 comparing the assertions in the collection of cited assertions of the cited document containing the cited content, and the assertions in the collection of citing assertions to identify any significant conflicts between the collection of cited assertions of the cited document and the collection of citing assertions.   
     
     
         30 . The non-transitory computer-readable medium of  claim 27 , wherein the operation further comprises scheduling each of the one or more cited references for a subsequent re-evaluation at a designated time. 
     
     
         31 . The non-transitory computer-readable medium of  claim 27 , wherein a given one of the cited documents comprises a cited research paper not written by an author of the primary research paper, wherein the operation further comprises:
 at the designated time, for the given one of the cited documents, re-evaluating the cited reference using the set of annotators to derive a third collection of one or more assertions regarding the cited reference;   comparing the assertions in the first collection of assertions and the assertions in the third collection of assertions to identify any conflicts between the first collection of assertions and the third collection of assertions; and   in response to identifying a conflict and without requiring any user input, programmatically notifying the author of the primary research document that the cited research paper has changed.   
     
     
         32 . A system to monitor documents in a managed research domain to determine whether changes in cited content undermine semantic conformity with citing content, the system comprising:
 one or more computer processors;   a storage repository; and   a memory storing a program which, when executed on the one or more computer processors, performs an operation comprising:
 extracting one or more citations from a primary research document, wherein each extracted citation references a respective one of one or more cited documents; 
 for each cited document, evaluating content of the cited document using a set of one or more annotators to derive a respective collection of cited assertions from the content of the cited document; 
 evaluating content of the primary research document using the set of one or more annotators to derive a collection of citing assertions from the content of the primary research document; 
 subsequent to storing the primary research document, each collection of cited assertions, and the collection of citing assertions in the storage repository, monitoring for changes to cited content in each cited document, including changes to cited content a first cited document; 
 for each monitored change to the cited content in the first cited document, determining, based on the derived collections of cited assertions and citing assertions, whether the respective monitored change undermines semantic conformity of the cited content in the first cited document with corresponding, citing content in the primary research document; and 
 upon determining that a first of the monitored changes to the cited content in the first cited document undermines semantic conformity with the citing content, designating the citing content as potentially lacking semantic conformity with the changed, cited content, wherein an indication of the citing content as being designated as potentially lacking semantic conformity is output. 
   
     
     
         33 . The system of  claim 32 , wherein the citing content is designated as potentially lacking semantic conformity with the changed, cited content, by annotating the citing content in the primary research document without annotating the changed, cited content in the first cited document. 
     
     
         34 . The system of  claim 33 , wherein the operation further comprises:
 in response to the designation, generating an alert identifying the citing content and indicating that semantic conformity is undermined;   outputting the alert to an author of the primary research document in order to prompt the author to reapprove at least the citing content in the primary research document in light of the changed, cited content in the first cited document; and   removing the designation from the primary research document after receiving, from the author, an indication of reapproval of the citing content in the primary research document.   
     
     
         35 . The system of  claim 34 , wherein at least one of the set of annotators normalizes terms in the primary research document and terms in the one or more cited documents to the common ontology, wherein the author is alerted of the annotation, wherein determining that the first monitored change to the cited content undermines semantic conformity with the citing content comprises:
 comparing the assertions in the collection of cited assertions of the cited document containing the cited content, and the assertions in the collection of citing assertions to identify any significant conflicts between the collection of cited assertions of the cited document and the collection of citing assertions.   
     
     
         36 . The system of  claim 35 , wherein the indication specifies approval of at least one of: (i) the citing content unaltered in the primary research document in light of the first monitored change; and (ii) an updated version of the citing content in light of the first monitored change; wherein the operation further comprises scheduling each of the one or more cited references for a subsequent re-evaluation at a designated time. 
     
     
         37 . The system of  claim 36 , wherein a given one of the cited documents comprises a cited research paper not written by the author of the primary research paper, wherein the operation further comprises:
 at the designated time, for each cited document, re-evaluating the cited reference using the set of annotators to derive a third collection of one or more assertions regarding the cited reference;   comparing the assertions in the first collection of assertions and the assertions in the third collection of assertions to identify any conflicts between the first collection of assertions and the third collection of assertions; and   in response to identifying a conflict and without requiring any user input, programmatically notifying the author of the primary research document that the cited research paper has changed.   
     
     
         38 . The system of  claim 37 , wherein the set of annotators are composed according to an Unstructured Information Management Architecture (UIMA), wherein the operation further comprises notifying the author of the cited research paper without requiring any user input specifying to notify the author of the cited research paper. 
     
     
         39 . The system of  claim 38 , wherein each annotator comprises an annotator specific to the managed research domain, wherein the common ontology is specific to the managed research domain, wherein the alert further identifies the cited content, wherein that the first monitored change to the cited content undermines the measure of semantic conformity with the citing content is determined without requiring any user input. 
     
     
         40 . The system of  claim 39 , wherein the program comprise a plurality of components including an annotator creation tool, the set of annotators, an annotator execution engine, and a document evaluation tool, the set of annotators including the set of annotators;
 wherein each cited document comprises a respective cited research paper, wherein the first and second collections of assertions includes each of: an assertion of causation, correlation, existence, truth, and falsity, respectively, relative to a concept from the common ontology; wherein each assertion is identified by a respective one of the one or more annotators.   
     
     
         41 . The system of  claim 40 , wherein each of the one or more annotators provide executable or interpretable code that creates annotations describing document content;
 wherein the indication specifies approval of, in respective instances: (i) the citing content unaltered in the primary research document in light of the first monitored change; and (ii) the updated version of the citing content in light of the first monitored change.   
     
     
         42 . The system of  claim 41 , wherein the set of annotators includes:
 (i) a normalization annotator normalizing the terms in the primary research document and the terms in the one or more cited documents to predefined terms used in a common ontology specific to the managed research domain;   (ii) a sentence annotator identifying each sentence in a given document selected from: the primary research document and the one or more cited documents;   (iii) a parts-of-speech annotator tagging words in each identified sentence and according to parts-of-speech;   (iv) a support annotator identifying references to any documents and further identifying which assertions are supported by the identified references; and   (v) a disambiguation annotator evaluating and reconciling results of at least two of the set of annotators.