Machine grading of short answers with explanations

Abstract

An example method embodying the disclosed technology comprises: digitally storing Teacher models and a Student model at a server computer; training each model with a corpus of unlabeled training data using Masked Language Modeling; fine-tuning each Teacher model for an ASAG task with labeled ground truth data; executing each Teacher model to generate and digitally store a respective set of class probabilities on an unlabeled task-specific data set for the ASAG task; further training the Student model by a linear ensemble of the Teacher models using KD; receiving, at the server computer, digital input comprising a target response text and a corresponding target reference answer text; programmatically inputting the target response text and the corresponding target reference answer text to the Student model, thereby outputting a corresponding predicted binary label; displaying correction data indicating the corresponding predicted binary label in a GUI; and, optionally, displaying explainability data in the GUI.

What is claimed is: 
     
       1. A computer-implemented method comprising:
 digitally storing, in memory of a server computer, a plurality of machine learning models, the plurality of machine learning models comprising a plurality of Teacher models and a Student model, each of the machine learning models comprising a multi-layer bidirectional Transformer encoder and having been trained with at least one corpus of unlabeled training data using Masked Language Modeling; 
 updating, in the memory of the server computer, each Teacher model by further programmatically training that Teacher model to perform an Automatic Short Answer Grading task with a labeled ground truth data set, the labeled ground truth data set comprising a plurality of data triplets, each data triplet comprising a response text, a corresponding reference answer text, and a corresponding binary label; 
 executing each of the Teacher models to cause programmatically generating and storing, in the memory of the server computer, a respective set of class probabilities on an unlabeled task-specific data set for the Automatic Short Answer Grading task; 
 updating, in the memory of the server computer, the Student model by further programmatically training the Student model, with the unlabeled task-specific data set, to minimize an error between predictions of the Student model and predictions of a linear ensemble of the Teacher models; 
 receiving, at the server computer, digital input comprising a target response text and a corresponding target reference answer text; 
 programmatically inputting the target response text and the corresponding target reference answer text to the Student model, thereby outputting a corresponding predicted binary label; 
 causing to be displayed, in a graphical user interface displayed on a device display of a client computing device, correction data indicating the corresponding predicted binary label. 
 
     
     
       2. The computer implemented method of  claim 1 , the plurality of Teacher models being programmed as a ROBERTa model, a Sentence-BERT model, and a Universal Sentence Encoder model, and the Student model being programmed as one of a Mobile BERT model, a SmallBERT model, or a MiniBERT model. 
     
     
       3. The computer-implemented method of  claim 1 , further comprising transmitting, from the server computer to the client computing device, first display instructions that are formatted to cause displaying, in the graphical user interface, the target response text and the corresponding target reference answer text with the correction data indicating the corresponding predicted binary label. 
     
     
       4. The computer-implemented method of  claim 3 , each of the target response text and the corresponding target reference answer text comprising digital data representing one or more words, and a respective token being used to represent, in the memory of the server computer, each word represented in the digital data. 
     
     
       5. The computer-implemented method of  claim 4 , further comprising transmitting, from the server computer to the client computing device, second display instructions that are formatted to cause indicating, in the graphical user interface, a sequence of one or more key words represented in at least one of the target response text or the corresponding target reference answer text that contributed most to the Student model programmatically determining the corresponding predicted binary label. 
     
     
       6. The computer-implemented method of  claim 5 , further comprising executing instructions implementing Integrated Gradients to programmatically compute an attribution score for each token based on a corresponding set of programmatically determined gradients of the predicted binary label with respect to each token, and the second display instructions being formatted to cause the indicating based on the computed attribution scores. 
     
     
       7. The computer-implemented method of  claim 6 , further comprising transmitting, from the server computer to the client computing device, third display instructions that are formatted to cause displaying, in the graphical user interface of the client computing device, highlighting on each word represented in each of the target response text and the corresponding target reference answer text caused to be displayed in the graphical user interface, each word being attributed a positive attribution score being highlighted, within a first color gradient, with a first level of highlighting corresponding to a magnitude of the positive attribution score, and each word being attributed a negative attribution score being highlighted, within a second color gradient, with a second level of highlighting corresponding to a magnitude of the negative attribution score. 
     
     
       8. The computer-implemented method of  claim 5 , further comprising transmitting, from the server computer to the client computing device, fourth display instructions that are formatted to cause displaying, in the graphical user interface of the client computing device, a grade representing a computed probability associated with the corresponding predicted binary label. 
     
     
       9. The computer-implemented method of  claim 8 , the correction data caused to be displayed in the graphical user interface further indicating if the target response text is correct or incorrect based on whether the computed probability associated with the corresponding predicted binary label exceeds a threshold probability stored in the memory of the server computer. 
     
     
       10. The computer-implemented method of  claim 9 , further comprising executing instructions programmed to:
 determine that the target response text is incorrect because the computed probability associated with the corresponding predicted binary label did not exceed the threshold probability stored in the memory of the server computer; 
 identify a set of phrases, each phrase being a unique sequence of one or more words being sequentially represented in the corresponding target reference answer text but not being sequentially represented in the target response text; and 
 identify the sequence of one or more key words that contributed most to the Student model programmatically determining the corresponding predicted binary label by executing instructions implementing a Perturbation technique to select the phrase of the set of phrases the position-wise inclusion of which in the target response answer text would have most increased the computed probability associated with the corresponding predicted binary label. 
 
     
     
       11. A computer system comprising:
 one or more processors; 
 digital electronic memory coupled to the one or more processors and storing one or more sequences of stored program instructions which, when executed by the one or more processors, cause the one or more processors to execute: 
 digitally storing, in memory of a server computer, a plurality of machine learning models, the plurality of machine learning models comprising a plurality of Teacher models and a Student model, each of the machine learning models comprising a multi-layer bidirectional Transformer encoder and having been trained with at least one corpus of unlabeled training data using Masked Language Modeling; 
 updating, in the memory of the server computer, each Teacher model by further programmatically training that Teacher model to perform an Automatic Short Answer Grading task with a labeled ground truth data set, the labeled ground truth data set comprising a plurality of data triplets, each data triplet comprising a response text, a corresponding reference answer text, and a corresponding binary label; 
 executing each of the Teacher models to cause programmatically generating and storing, in the memory of the server computer, a respective set of class probabilities on an unlabeled task-specific data set for the Automatic Short Answer Grading task; 
 updating, in the memory of the server computer, the Student model by further programmatically training the Student model, with the unlabeled task-specific data set, to minimize an error between predictions of the Student model and predictions of a linear ensemble of the Teacher models; 
 receiving, at the server computer, digital input comprising a target response text and a corresponding target reference answer text; 
 programmatically inputting the target response text and the corresponding target reference answer text to the Student model, thereby outputting a corresponding predicted binary label; 
 causing to be displayed, in a graphical user interface displayed on a device display of a client computing device, correction data indicating the corresponding predicted binary label. 
 
     
     
       12. The system of  claim 11 , the plurality of Teacher models being programmed as a ROBERTa model, a Sentence-BERT model, and a Universal Sentence Encoder model, and the Student model being programmed as one of a Mobile BERT model, a SmallBERT model, or a MiniBERT model. 
     
     
       13. The system of  claim 11 , the instructions further executable to cause performance of transmitting, from the server computer to the client computing device, first display instructions that are formatted to cause displaying, in the graphical user interface, the target response text and the corresponding target reference answer text with the correction data indicating the corresponding predicted binary label. 
     
     
       14. The system of  claim 13 , each of the target response text and the corresponding target reference answer text comprising digital data representing one or more words, and a respective token being used to represent, in the memory of the server computer, each word represented in the digital data. 
     
     
       15. The system of  claim 14 , the instructions further executable to cause performance of transmitting, from the server computer to the client computing device, second display instructions that are formatted to cause indicating, in the graphical user interface, a sequence of one or more key words represented in at least one of the target response text or the corresponding target reference answer text that contributed most to the Student model programmatically determining the corresponding predicted binary label. 
     
     
       16. The system of  claim 15 , the instructions further executable to cause performance of executing instructions implementing Integrated Gradients to programmatically compute an attribution score for each token based on a corresponding set of programmatically determined gradients of the predicted binary label with respect to each token, and the second display instructions being formatted to cause the indicating based on the computed attribution scores. 
     
     
       17. The system of  claim 16 , the instructions further executable to cause performance of transmitting, from the server computer to the client computing device, third display instructions that are formatted to cause displaying, in the graphical user interface of the client computing device, highlighting on each word represented in each of the target response text and the corresponding target reference answer text caused to be displayed in the graphical user interface, each word being attributed a positive attribution score being highlighted, within a first color gradient, with a first level of highlighting corresponding to a magnitude of the positive attribution score, and each word being attributed a negative attribution score being highlighted, within a second color gradient, with a second level of highlighting corresponding to a magnitude of the negative attribution score. 
     
     
       18. The system of  claim 15 , the instructions further executable to cause performance of transmitting, from the server computer to the client computing device, fourth display instructions that are formatted to cause displaying, in the graphical user interface of the client computing device, a grade representing a computed probability associated with the corresponding predicted binary label. 
     
     
       19. The system of  claim 18 , the correction data caused to be displayed in the graphical user interface further indicating if the target response text is correct or incorrect based on whether the computed probability associated with the corresponding predicted binary label exceeds a threshold probability stored in the memory of the server computer. 
     
     
       20. The system of  claim 19 , the instructions further executable to cause performance of:
 determining that the target response text is incorrect because the computed probability associated with the corresponding predicted binary label did not exceed the threshold probability stored in the memory of the server computer; 
 identifying a set of phrases, each phrase being a unique sequence of one or more words being sequentially represented in the corresponding target reference answer text but not being sequentially represented in the target response text; and 
 identifying the sequence of one or more key words that contributed most to the Student model programmatically determining the corresponding predicted binary label by executing instructions implementing a Perturbation technique to select the phrase of the set of phrases the position-wise inclusion of which in the target response answer text would have most increased the computed probability associated with the corresponding predicted binary label.