Model training method and apparatus

Abstract

A model training method and apparatus is disclosed, where the model training method acquires first output data of a student model for first input data and second output data of a teacher model for second input data and trains the student model such that the first output data and the second output data are not distinguished from each other. The student model and the teacher model have different structures.

What is claimed is: 
     
       1. A method of training a model, the method comprising:
 acquiring first output data of a student model for first input data and second output data of a teacher model for second input data; and 
 training the student model such that the first output data and the second output data are not distinguished from each other, 
 wherein the student model and the teacher model have different structures, and 
 wherein the first input data is one of text data and speech data and the second input data is the other of the text data and speech data. 
 
     
     
       2. The method of  claim 1 , wherein the student model and the teacher model are configured to process different tasks. 
     
     
       3. The method of  claim 1 , wherein the first input data and the second input data are different types of data. 
     
     
       4. The method of  claim 1 , wherein the first input data and the second input data are unlabeled data. 
     
     
       5. The method of  claim 1 , wherein the training of the student model comprises:
 training the student model such that the first output data and the second output data are not distinguished from each other by a discriminator model and 
 the discriminator model is configured to distinguish between the first output data and the second output data. 
 
     
     
       6. The method of  claim 1 , wherein the first output data and the second output data are a same type of data. 
     
     
       7. The method of  claim 1 , wherein the student model is a speech recognition model and
 the teacher model is a language model that outputs text data based on an expression of a domain. 
 
     
     
       8. The method of  claim 1 , wherein the first input data, the second input data, the first output data, and the second output data are sequence data. 
     
     
       9. The method of  claim 1 , wherein the training of the student model comprises:
 determining an adversarial loss based on a degree to which the first output data and the second output data are distinguished from each other and training the student model to reduce the adversarial loss. 
 
     
     
       10. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of  claim 1 . 
     
     
       11. An apparatus for training a model, the apparatus comprising;
 a processor configured to acquire first output data of a student model for first input data and second output data of a teacher model for second input data and to train the student model such that the first output data and the second output data are not distinguished from each other; and 
 a memory configured to store a parameter of the student model, 
 wherein the student model and the teacher model have different structures, and 
 wherein the first input data is one of text data and speech data and the second input data is the other of the text data and speech data. 
 
     
     
       12. The apparatus of  claim 11 , wherein the student model and the teacher model are configured to process different tasks. 
     
     
       13. The apparatus of  claim 11 , wherein the first input data and the second input data are different types of data. 
     
     
       14. The apparatus of  claim 11 , wherein the first input data and the second input data are unlabeled data. 
     
     
       15. The apparatus of  claim 11 , wherein the processor is further configured to train the student model such that the first output data and the second output data are not distinguished from each other by a discriminator model, and
 the discriminator model is configured to distinguish between the first output data and the second output data. 
 
     
     
       16. The apparatus of  claim 11 , wherein the first output data and the second output data are a same type of data. 
     
     
       17. The apparatus of  claim 11 , wherein the student model is a speech recognition model, and
 the teacher model is a language model that outputs text data based on an expression of a domain. 
 
     
     
       18. An apparatus for training a model, the apparatus comprising;
 a memory configured to store a student model, a teacher model, and a discriminator model; and 
 a processor configured to
 acquire first output data of the student model for first input data and second output data of the teacher model for second input data, 
 train the discriminator model to distinguish between the first output data and the second output data, and 
 train the student model to minimize a distinction between the first output data and the second output data at the discriminator mode, 
 
 wherein the first input data is one of text data and speech data and the second input data is the other of the text data and speech data. 
 
     
     
       19. The apparatus of  claim 18 , wherein a number of hidden layers of the student model is lesser than a number of hidden layers of the teacher model. 
     
     
       20. The apparatus of  claim 18 , wherein the first input data and the second output data comprise unlabeled data.