Interface for artificial intelligence training

Abstract

Media and method for a user interface for training an artificial intelligence system. Many artificial intelligence systems require large volumes of labeled training data before they can accurately classify previously unseen data items. However, for some problem domains, no pre-labeled training data set may be available. Manually labeling training data sets by a subject-matter expert is a laborious process. An interface to enable such a subject-matter expert to accurately, consistently, and quickly label training data sets is disclosed herein. By allowing the subject-matter expert to easily navigate between training data items and select the applicable labels, operation of the computer is improved.

Having thus described various embodiments of the invention, what is claimed as new and desired to be protected by Letters Patent includes the following: 
     
         1 . A method of training an artificial intelligence model, the method comprising:
 ingesting a plurality of training data items;   determining a plurality of applicable labels within the plurality of training data items;   generating for display, to a user within a user interface, a plurality of indicia corresponding to a plurality of labels, wherein each label of the plurality of labels is applicable to one or more items of the plurality of training data items;   receiving, from the user through the user interface, a selection of one or more selected indicia from the plurality of indicia;   responsive to the selection of the one or more selected indicia, generating a classification for the one or more items of the plurality of training data items based on the one or more selected indicia;   storing, with one or more respective items from the plurality of training data items, the one or more selected indicia within a data store; and   training the artificial intelligence model based on the one or more respective items, the one or more selected indicia, and the classification for the one or more items.   
     
     
         2 . The method of  claim 1 , further comprising:
 classifying, using the artificial intelligence model, one or more real-time data items.   
     
     
         3 . The method of  claim 2 , further comprising:
 retraining the artificial intelligence model based at least in part on the one or more real-time data items.   
     
     
         4 . The method of  claim 1 , further comprising:
 receiving, from the user through the user interface, a subsequent selection corresponding to a misclassified data item of the plurality of training data items;   responsive to the subsequent selection, relabeling the misclassified data item; and   retraining the artificial intelligence model based at least in part on the misclassified data item.   
     
     
         5 . The method of  claim 1 , further comprising:
 automatically preselecting one or more labels of the plurality of labels using the artificial intelligence model.   
     
     
         6 . The method of  claim 5 , further comprising:
 responsive to the selection from the user, automatically deselecting the one or more labels that were preselected by the artificial intelligence model.   
     
     
         7 . The method of  claim 1 , further comprising:
 responsive to the selection of the one or more selected indicia, updating the display of the user interface with the one or more selected indicia.   
     
     
         8 . One or more non-transitory computer-readable media that store computer-executable instructions that, when executed by at least one processor, perform a method of training an artificial intelligence model, the method comprising:
 receiving a plurality of training data items from a training data store;   ingesting the plurality of training data items;   determining a plurality of applicable labels within the plurality of training data items;   generating for display, to a user within a user interface, a plurality of indicia corresponding to a plurality of labels, wherein each label of the plurality of labels is applicable to one or more items of the plurality of training data items;   receiving, from the user through the user interface, a selection of one or more selected indicia from the plurality of indicia;   responsive to the selection of the one or more selected indicia, generating a classification for the one or more items of the plurality of training data items based on the one or more selected indicia;   storing, with one or more respective items from the plurality of training data items, the one or more selected indicia within a data store; and   training the artificial intelligence model based on the one or more respective items, the one or more selected indicia, and the classification for the one or more items.   
     
     
         9 . The one or more non-transitory computer-readable media of  claim 8 , wherein the user interface comprises a first pane displaying:
 at least a subset of the plurality of training data items; and   an indication of a current training data item of the plurality of training data items.   
     
     
         10 . The one or more non-transitory computer-readable media of  claim 9 , wherein the user interface further comprises a second pane displaying:
 the plurality of indicia corresponding to the plurality of labels.   
     
     
         11 . The one or more non-transitory computer-readable media of  claim 10 ,
 wherein the user interface is configured such that the user can select and unselect one or more labels of the plurality of labels,   wherein, by selecting the one or more labels of the plurality of labels, the user indicates that the artificial intelligence model should apply the one or more labels when classifying the current training data item,   wherein automatic preliminary classification of the plurality of training data items occurs in real time based on a current state of training of the artificial intelligence model,   wherein the artificial intelligence model filters out one or more filtered labels of the plurality of labels that are not applicable to the current training data item based on the automatic preliminary classification.   
     
     
         12 . The one or more non-transitory computer-readable media of  claim 11 , wherein the second pane further displays:
 a toggle,   wherein the artificial intelligence model sorts the one or more labels based on an estimated likelihood of the one or more labels being applicable to the current training data item and preselects one or more preselected labels of the plurality of labels as applicable to the current training data item based on the automatic preliminary classification,   wherein the automatic preliminary classification of subsequent data items is updated based on the user selecting labels and unselecting preselected labels corresponding to the current training data item;   a not-applicable indicium to indicate that at least one of the plurality of training data items is not relevant to any of the one or more labels; and   a function for displaying indicia corresponding to all available labels, when the user determines an applicable label has been incorrectly filtered out based on the automatic preliminary classification.   
     
     
         13 . The one or more non-transitory computer-readable media of  claim 12 , wherein the second pane further displays:
 a forward button to allow the user to move from the current training data item to a next one of the plurality of training data items; and   a back button to allow the user to move from the current training data item to a previous one of the plurality of training data items.   
     
     
         14 . The one or more non-transitory computer-readable media of  claim 8 , wherein the plurality of indicia comprises at least one icon corresponding to a respective label of the plurality of labels. 
     
     
         15 . A system comprising:
 at least one processor; and   one or more non-transitory computer-readable media that store computer-executable instructions that, when executed by the at least one processor, perform a method of training an artificial intelligence model, the method comprising:
 receiving a plurality of training data items from a training data store; 
 ingesting the plurality of training data items; 
 determining a plurality of applicable labels within the plurality of training data items; 
 generating for display, to a user within a user interface, a plurality of indicia corresponding to a plurality of labels, wherein each label of the plurality of labels is applicable to one or more items of the plurality of training data items; 
 receiving, from the user through the user interface, a selection of one or more selected indicia from the plurality of indicia; 
 responsive to the selection of the one or more selected indicia, generating a classification for the one or more items of the plurality of training data items based on the one or more selected indicia; and 
 storing, with one or more respective items from the plurality of training data items, the one or more selected indicia within a data store. 
   
     
     
         16 . The system of  claim 15 , wherein the method further comprises:
 training the artificial intelligence model based on the one or more respective items, the one or more selected indicia, and the classification for the one or more items.   
     
     
         17 . The system of  claim 16 , wherein an automatic preliminary classification of the plurality of training data items occurs in real time based on a current state of training of the artificial intelligence model. 
     
     
         18 . The system of  claim 16 , wherein each training data item of the plurality of training data items comprises a statement about a taxpayer's financial situation. 
     
     
         19 . The system of  claim 18 , wherein each label of the plurality of labels corresponds to a tax topic. 
     
     
         20 . The system of  claim 19 , wherein the user is a tax professional and the user interface is generated for display on a user device of the tax professional.