Graphical systems and methods for human-in-the-loop machine intelligence

Abstract

Methods and systems are disclosed for creating and linking a series of interfaces configured to display information and receive confirmation of classifications made by a natural language modeling engine to improve organization of a collection of documents into an hierarchical structure. In some embodiments, the interfaces may display to an annotator a plurality of labels of potential classifications for a document as identified by a natural language modeling engine, collect annotated responses from the annotator, aggregate the annotated responses across other annotators, analyze the accuracy of the natural language modeling engine based on the aggregated annotated responses, and predict accuracies of the natural language modeling engine's classifications of the documents.

What is claimed is: 
     
         1 . A method comprising:
 accessing first inputs managed through a first graphic user interface operated by a project manager, wherein accessing first inputs comprises:
 accessing at least one document sourced by a third party user; 
 accessing at least one first label associated with the at least one document, wherein the at least one first label is selected from a classification ontology built from a collection comprising the at least one document; and 
 accessing a plurality of first guidelines describing at least one first label; 
   accessing a second input from a second graphic user interface operated by an expert annotator, wherein the second input is at least one of the group comprising at least one second label and at least one second guideline;   constructing a work unit interface, wherein constructing a work unit interface comprises:
 assigning a document among the at least one document to a document pane within the work unit interface; 
 assigning the at least one first label or the at least one second label to a label pane within the work unit interface; 
 generating a human readable prompt requesting a task of the document, wherein the task is a confirmation of the accuracy for classification of a label for the document within the classification ontology; 
 assigning the generated human readable prompt to a prompt pane within the work unit interface; 
 pairing a single first guideline from the plurality of first guidelines or a single second guideline with a single label of the plurality of first labels or single second label; 
 creating a reference button for each single guideline paired with a single label; and 
 placing the reference button adjacent to the paired label within the label pane of the work unit interface; 
   displaying to at least one annotator operating a third graphic user interface or at least one expert annotator operating the second graphic user interface the work unit interface comprising each of the document, the human readable prompt, the at least one first label or the at least one second label, and the reference button for each single guideline;   receiving at least one annotation on the work unit interface displayed on the third graphic user interface or the second graphic user interface; and   aggregating the at least one annotation received on the first graphic user interface operated by the project manager.   
     
     
         2 . The method of  claim 1 , wherein aggregating the at least one annotation further comprises:
 constructing an annotation agreement interface, wherein constructing an annotation agreement interface further comprises:
 creating a label feedback pane, wherein the label feedback pane is configured to display at least one from the group comprising a plurality of label panes and each label pane comprising a description of a label, an indicator of the number of annotated documents associated with the label, a button to delete the label from the classification ontology, and a button to edit the label; 
 creating a learning curve pane, wherein the learning curve pane is configured to display a graphical representation of the relationship among a number of annotations aggregated on the first graphic user interface, an agreement among annotators of a label or task of a document, and an accuracy of the classification ontology; 
 creating an annotation feedback pane, wherein the annotation feedback pane is configured to display at least one of the group comprising a collection annotation agreement score, an individual annotator agreement list, a suggested label collapse list, an agreement per label graphical representation, a collapsed agreement per label graphical representation, and a per document agreement list; and 
 creating a rules pane, wherein the rules pane is configured to display at least one of the group comprising a phrase pane, a weighting adjustment pane, and an add rule pane. 
   
     
     
         3 . The method of  claim 2 , further comprising computing a plurality of initial annotation agreements from the aggregated annotations. 
     
     
         4 . The method of  claim 3 , further comprising populating the annotation agreement interface's label feedback pane, learning curve pane, and annotation feedback pane with computed initial annotation agreements. 
     
     
         5 . The method of  claim 4 , further comprising displaying the plurality of initial annotation agreements on the annotation agreement interface of the first graphic user interface. 
     
     
         6 . The method of  claim 5 , further comprising:
 receiving a request through the annotation agreement interface of the first graphic user interface to collapse at least one first label or task into at least one second label or task;   preparing at least one subsequent annotation agreement as computed from the initial annotation agreement of the first label or task and second label or task collapsed into one another; and   displaying on the annotation agreement interface of the first graphic user interface the at least one subsequent annotation agreement among the plurality of initial annotation agreements.   
     
     
         7 . The method of  claim 6 , further comprising:
 accessing at least one revised guideline through the first graphic user interface or second graphic user interface;   unpairing at least one single guideline paired with a single label on the work unit interface;   pairing the at least one revised guideline with an unpaired single label on the work unit interface;   creating a second reference button for the at least one paired revised guideline; and   displaying the second reference button within the work unit interface's label pane adjacent to the single label.   
     
     
         8 . A non-transitory computer readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations comprising:
 accessing first inputs managed through a first graphic user interface operated by a project manager, wherein accessing first inputs comprises;
 accessing at least one document sourced by a third party user; 
 accessing at least one first label associated with the at least one document, wherein the at least one first label is selected from a classification ontology built from a collection comprising the at least one document; and 
 accessing a plurality of first guidelines describing at least one first label; 
   accessing a second input from a second graphic user interface operated by an expert annotator, wherein the second input is at least one of the group comprising at least one second label and at least one second guideline;   constructing a work unit interface, wherein constructing a work unit interface comprises:
 assigning a document among the at least one document to a document pane within the work unit interface; 
 assigning the at least one first label or the at least one second label to a label pane within the work unit interface; 
 generating a human readable prompt requesting a task of the document, wherein the task is a confirmation of the accuracy for classification of a label for the document within the classification ontology; 
 assigning the generated human readable prompt to a prompt pane within the work unit interface; 
 pairing a single first guideline from the plurality of first guidelines or a single second guideline with a single label of the plurality of first labels or single second label; 
 creating a reference button for each single guideline paired with a single label; and 
 placing the reference button adjacent to the paired label within the label pane of the work unit interface; 
   displaying to at least one annotator operating a third graphic user interface or at least one expert annotator operating the second graphic user interface the work unit interface comprising each of the document, the human readable prompt, the at least one first label or the at least one second label, and the reference button for each single guideline;   receiving at least one annotation on the work unit interface displayed on the third graphic user interface or the second graphic user interface; and   aggregating the at least one annotation received on the first graphic user interface operated by the project manager.   
     
     
         9 . The computer readable medium of  claim 8 , wherein the operations to aggregate the at least one annotation further comprise:
 constructing an annotation agreement interface, wherein constructing an annotation agreement interface further comprises:
 creating a label feedback pane, wherein the label feedback pane is configured to display at least one from the group comprising a plurality of label panes and each label pane comprising a description of a label, an indicator of the number of annotated documents associated with the label, a button to delete the label from the classification ontology, and a button to edit the label; 
 creating a learning curve pane, wherein the learning curve pane is configured to display a graphical representation of the relationship among a number of annotations aggregated on the first graphic user interface, an agreement among annotators of a label or task of a document, and an accuracy of the classification ontology; 
 creating an annotation feedback pane, wherein the annotation feedback pane is configured to display at least one of the group comprising a collection annotation agreement score, an individual annotator agreement list, a suggested label collapse list, an agreement per label graphical representation, a collapsed agreement per label graphical representation, and a per document agreement list; and 
 creating a rules pane, wherein the rules pane is configured to display at least one of the group comprising a phrase pane, a weighting adjustment pane, and an add rule pane. 
   
     
     
         10 . The computer readable medium of  claim 9 , wherein the operations further comprise:
 computing a plurality of initial annotation agreements from the aggregated annotations.   
     
     
         11 . The computer readable medium of  claim 10 , wherein the operations further comprise:
 populating the annotation agreement interface's label feedback pane, learning curve pane, and annotation feedback pane with computed initial annotation agreements.   
     
     
         12 . The computer readable medium of  claim 11 , wherein the operations further comprise:
 displaying the plurality of initial annotation agreements on the annotation agreement interface of the first graphic user interface.   
     
     
         13 . The computer readable medium of  claim 12 , wherein the operations further comprise:
 receiving a request through the annotation agreement interface of the first graphic user interface to collapse at least one first label or task into at least one second label or task;   preparing at least one subsequent annotation agreement as computed from the initial annotation agreement of the first label or task and second label or task collapsed into one another; and   displaying on the annotation agreement interface of the first graphic user interface the at least one subsequent annotation agreement among the plurality of initial annotation agreements.   
     
     
         14 . The computer readable medium of  claim 13 , wherein the operations further comprise:
 accessing at least one revised guideline through the first graphic user interface or second graphic user interface;   unpairing at least one single guideline paired with a single label on the work unit interface;   pairing the at least one revised guideline with an unpaired single label on the work unit interface;   creating a second reference button for the at least one paired revised guideline; and   
       displaying the second reference button within the work unit interface's label pane adjacent to the single label. 
     
     
         15 . An interface integration system comprising:
 a data processor;   an input and output module from at least one of the group comprising a first graphic user interface associated with a project manager, a second graphic user interface operated by an expert annotator, and a third graphic user interface operated by an annotator;   a natural language modeling engine operably coupled to the input and output module, configured to execute instructions received from the data processor to:
 access first inputs managed by the first graphic user interface, wherein the access to first inputs further comprises;
 access at least one document sourced by a third party user; 
 access at least one first label associated with the at least one document, wherein the at least one first label is selected from a classification ontology built from a collection comprising the at least one document; and 
 access a plurality of first guidelines describing at least one first label; 
 
 access a second input from a second graphic user interface, wherein the second input is at least one of the group comprising at least one second label and at least one second guideline; 
 construct a work unit interface, wherein to construct a work unit interface the natural language modeling engine is further configured to:
 assign a document among the at least one document to a document pane within the work unit interface; 
 assign the at least one first label or the at least one second label to a label pane within the work unit interface; 
 generate a human readable prompt requesting a task of the document, wherein the task is a confirmation of the accuracy for classification of a label for the document within the classification ontology; 
 assign the generated human readable prompt to a prompt pane within the work unit interface; 
 pair a single first guideline from the plurality of first guidelines or a single second guideline with a single label of the plurality of first labels or single second label; 
 create a reference button for each single guideline paired with a single label; and 
 place the reference button adjacent to the paired label within the label pane of the work unit interface; 
 
 display to at least one annotator operating a third graphic user interface or at least one expert annotator operating the second graphic user interface the work unit interface comprising each of the document, the human readable prompt, the at least one first label or the at least one second label, and the reference button for each single guideline; 
 receive at least one annotation on the work unit interface displayed on the third graphic user interface; and 
 aggregate the at least one annotation in an annotation agreement interface of the first graphic user interface associated with a project manager. 
   
     
     
         16 . The interface integration system of  claim 15 , wherein the natural language modeling engine operably coupled to the input and output module is further configured to execute instructions received from the data processor to:
 construct an annotation agreement interface, wherein the instructions to construct an annotation agreement interface further comprises instructions to:
 create a label feedback pane, wherein the instructions to create a label feedback pane configure the display of at least one from the group comprising a plurality of label panes and each label pane comprising a description of a label, an indicator of the number of annotated documents associated with the label, a button to delete the label from the classification ontology, and a button to edit the label; 
 create a learning curve pane, wherein the instructions to create a learning curve pane configure the display of a graphical representation of the relationship among a number of annotations aggregated on the first graphic user interface, an agreement among annotators of a label or task of a document, and an accuracy of the classification ontology; 
 create an annotation feedback pane, wherein the instructions to create an annotation feedback pane configured the display of at least one of the group comprising a collection annotation agreement score, an individual annotator agreement list, a suggested label collapse list, an agreement per label graphical representation, a collapsed agreement per label graphical representation, and a per document agreement list; and 
 create a rules pane, wherein the instructions to create a rules pane configure the display of at least one of the group comprising a phrase pane, a weighting adjustment pane, and an add rule pane. 
   
     
     
         17 . The interface integration system of  claim 16 , wherein the natural language modeling engine operably coupled to the input and output module is further configured to execute instructions received from the data processor to:
 compute a plurality of initial annotation agreements from the aggregated annotations; and   populate the annotation agreement interface's label feedback pane, learning curve pane, and annotation feedback pane.   
     
     
         18 . The interface integration system of  claim 17 , wherein the natural language modeling engine operably coupled to the input and output module is further configured to execute instructions from the data processor to display the plurality of initial annotation agreements on the annotation agreement interface of the first graphic user interface. 
     
     
         19 . The interface integration system of  claim 18 , wherein the natural language modeling engine operably coupled to the input and output module is further configured to execute instructions received from the data processor to:
 receive a request through the annotation agreement interface of the first graphic user interface to collapse at least one first label or task into at least one second label or task;   prepare at least one subsequent annotation agreement as computed from the initial annotation agreement of the first label or task and second label or task collapsed into one another; and   display on the annotation agreement interface of the first graphic user interface the at least one subsequent annotation agreement among the plurality of initial annotation agreements.   
     
     
         20 . The interface integration system of  claim 19 , wherein the natural language modeling engine operably coupled to the input and output module is further configured to execute instructions received from the data processor to:
 access at least one revised guideline through the first graphic user interface or second graphic user interface;   unpair at least one single guideline paired with a single label on the work unit interface;   pair the at least one revised guideline with an unpaired single label on the work unit interface;   create a second reference button for the at least one paired revised guideline; and   display the second reference button within the work unit interface's label pane adjacent to the single label.