Systems, techniques, and interfaces for obtaining and annotating training instances

Abstract

A previously trained classification model associated with the machine learning system is configured to process an input to generate i) a first prediction that represents a characteristic associated with the input, and ii) a representation of accuracy associated with the prediction. A retraining subsystem is configured to receive the input, the first prediction, and the representation of accuracy. The retraining subsystem processes the input to generate a prediction representing a characteristic. A sufficiency of certainty of the first prediction is determined based on at least the input, the first prediction, the measure of accuracy, and the second prediction. Based at least on the determined sufficiency the retraining subsystem causes the machine learning system to be automatically retrained, be retrained using the input with active learning or not retrained.

1 .- 20 . (canceled) 
     
     
         21 . A method for operating a machine learning system, the method comprising:
 identifying an input;   
       processing the input using a model to generate i) a first prediction that represents a characteristic associated with the input, and ii) a representation of accuracy associated with the prediction;
 comparing the representation of accuracy to at least a first and a second threshold value; and 
 based on the comparison, performing one of an active learning and an auto labeling process using the input and the first prediction. 
 
     
     
         22 . The method of  claim 21 , wherein the comparison indicates that the representation of accuracy is less than the first threshold value and greater than the second threshold value, the method further comprising:
 performing an active learning process and causing information associated with the input and the first prediction to be displayed on a user interface; and   receiving one of an acceptance and a decline of the first prediction as an input in response to the display.   
     
     
         23 . The method of  claim 22 , wherein the input in response to the display is an acceptance, the method further comprising:
 saving the first prediction as a label associated with the input.   
     
     
         24 . The method of  claim 22 , wherein the input in response to the display is a decline, the method further comprising:
 reusing the input as a subsequent input to the machine learning system.   
     
     
         25 . The method of  claim 22 , wherein the input in response to the display is a decline, the method further comprising:
 identifying at least a second prediction that represents a characteristic associated with the input;   causing information associated with the input and the second prediction to be displayed on a user interface; and   receiving one of an acceptance and a decline of the second prediction as an input in response to the display.   
     
     
         26 . The method of  claim 25 , wherein the input in response to the display of the input and the second prediction is an acceptance, the method further comprising:
 saving the second prediction as a label associated with the input.   
     
     
         27 . The method of  claim 22 , wherein the input in response to the display is generated by at least one of (i) a swipe action on a touch screen display device, and (ii) a mouse selection on a computing device. 
     
     
         28 . The method of  claim 21 , wherein the representation of accuracy associated with the first prediction is a Softmax confidence value. 
     
     
         29 . The method of  claim 21 , wherein the comparison indicates that the representation of accuracy is greater than both the first threshold value and the second threshold value, the method further comprising:
 performing an auto labeling process and automatically associating the first prediction with the input as a label.   
     
     
         30 . The method of  claim 21 , wherein the comparison indicates that the representation of accuracy is less than both the first threshold value and the second threshold value, the method further comprising:
 discarding the input.   
     
     
         31 . A method for operating a machine learning system, the method comprising:
 identifying an input;
 processing the input using a model to generate i) a first prediction that represents a characteristic associated with the input, and ii) a representation of accuracy associated with the prediction; 
 comparing the representation of accuracy to a first threshold value and a second threshold value and, based at least in part on the comparison: 
 (i) in the event the representation of accuracy is greater than both the first and the second threshold value, automatically associating the first prediction with the input as a label; and 
 (ii) in the event the representation of accuracy is less than the first threshold value and greater than the second threshold value, causing information associated with the input and the first prediction to be displayed on a user interface, and receiving one of an acceptance and a decline of the first prediction as an input in response to the display. 
   
     
     
         32 . The method of  claim 31 , wherein the input in response to the display is an acceptance, the method further comprising:
 saving the first prediction as a label associated with the input.   
     
     
         33 . The method of  claim 31 , wherein the input in response to the display is a decline, the method further comprising:
 reusing the input as a subsequent input to the machine learning system.   
     
     
         34 . The method of  claim 31 , wherein the input in response to the display is a decline, the method further comprising:
 identifying at least a second prediction that represents a characteristic associated with the input;   causing information associated with the input and the second prediction to be displayed on a user interface; and   receiving one of an acceptance and a decline of the second prediction as an input in response to the display.   
     
     
         35 . The method of  claim 34 , wherein the input in response to the display of the input and the second prediction is an acceptance, the method further comprising:
 saving the second prediction as a label associated with the input.   
     
     
         36 . The method of  claim 31 , wherein the input in response to the display is generated by at least one of (i) a swipe action on a touch screen display device, and (ii) a mouse selection on a computing device. 
     
     
         37 . A system comprising:
 a processing unit; and   a memory storage device including program code that when executed by the processing unit causes to the system to:   identify an input;   process the input using a model to generate i) a first prediction that represents a characteristic associated with the input, and ii) a representation of accuracy associated with the prediction;   compare the representation of accuracy to at least a first and a second threshold value; and   based on the comparison, performing one of an active learning and an auto labeling process using the input and the first prediction.   
     
     
         38 . The system of  claim 37 , wherein the comparison indicates that the representation of accuracy is less than the first threshold value and greater than the second threshold value, the system further comprising code that when executed by the processing unit causes the system to:
 perform an active learning process and causing information associated with the input and the first prediction to be displayed on a user interface; and   receive one of an acceptance and a decline of the first prediction as an input in response to the display.   
     
     
         39 . The system of  claim 38 , wherein the input in response to the display is a decline, the system further comprising code that when executed by the processing unit causes the system to:
 identify at least a second prediction that represents a characteristic associated with the input;   cause information associated with the input and the second prediction to be displayed on a user interface; and
 receive one of an acceptance and a decline of the second prediction as an input in response to the display. 
   
     
     
         40 . The system of  claim 39 , wherein the input in response to the display of the input and the second prediction is an acceptance, the system further comprising code that when executed by the processing unit causes the system to:
 save the second prediction as a label associated with the input.