Generation of optimized knowledge-based language model through knowledge graph multi-alignment

Abstract

A language module is joint trained with a knowledge module for natural language understanding by aligning a first knowledge graph with a second knowledge graph. The knowledge module is trained on the aligned knowledge graphs. Then, the knowledge module is integrated with the language module to generate an integrated knowledge-language module.

What is claimed is: 
     
       1. A computer implemented method for joint training a language module with a knowledge module for natural language understanding, the method comprising:
 obtaining a first knowledge graph comprising a first set of entities and a first set of relations, each relation of the first set of relations describing a relationship between two or more entities included in the first set of entities; 
 obtaining a second knowledge graph comprising a second set of entities and a second set of relations, each relation of the second set of relations describing a relationship between two or more entities included in the second set of entities; 
 aligning the first knowledge graph and second knowledge graph such that a first subset of entities and relations from the first knowledge graph corresponds to a second subset of entities and relations from the second knowledge graph; 
 accessing a language module comprising a first language model and a second language model, wherein the first language model generates a first set of embeddings comprising initial entity embeddings corresponding to the first knowledge graph and a second set of embeddings comprising contextual representations; 
 accessing a knowledge module and training the knowledge module with embeddings from the first language model; 
 generating an integrated knowledge-language module by integrating the language module and the knowledge module by configuring the knowledge module to provide knowledge information input to the language module and by configuring the language module to provide context information input to the knowledge module; 
 further training the knowledge module by applying the aligned knowledge graphs to the knowledge module such that the integrated knowledge-language module is configured to perform semantic analysis for the entities and entity relations in the second knowledge graph based on knowledge learned from entities and entity relations in the first knowledge graph. 
 
     
     
       2. The method of  claim 1 , further comprising:
 generating the first knowledge graph in a first language; 
 generating the second knowledge graph in a second language; 
 configuring the knowledge module to generate knowledge-based entity representations in the second language based on the first knowledge graph; and 
 training the integrated knowledge-language module to perform semantic analysis in the second language based on knowledge learned from entities and entity relations in the first language. 
 
     
     
       3. The method of  claim 2 , further comprising:
 configuring the knowledge module to generate knowledge-based entity representations in the first language based on the second knowledge graph; and 
 training the integrated knowledge-language module to perform semantic analysis in the first language based on knowledge learned from entities and entity relations in the second language. 
 
     
     
       4. The method of  claim 2 , further comprising:
 obtaining electronic content comprising a first set of speech transcriptions in the first language; 
 applying the electronic content as input to the integrated knowledge-language module; and 
 translating the first set of speech transcriptions into a second set of speech transcriptions in the second language using the integrated knowledge-language module. 
 
     
     
       5. The method of  claim 1 , further comprising:
 generating the first knowledge graph as a domain-independent knowledge graph; 
 generating the second knowledge graph as an enterprise-specific knowledge graph, wherein the second set of entities and second set of relations correspond to an enterprise-specific domain; 
 adapting the knowledge module to generate enterprise-specific knowledge-based entity representations; and 
 adapting the integrated knowledge-language module to perform improved semantic analysis for enterprise-specific electronic language content. 
 
     
     
       6. The method of  claim 1 , further comprising:
 generating the first knowledge graph in a first domain; 
 generating the second knowledge graph in an entity-specific personalized domain; 
 adapting the knowledge module to generate knowledge-based entity representations in the entity-specific personalized domain based on the first knowledge graph; and 
 training the integrated knowledge-language module to perform semantic analysis in the entity-specific personalized domain based on personalized knowledge learned from entities and entity relations in the first domain. 
 
     
     
       7. The method of  claim 1 , further comprising:
 obtaining the first knowledge graph as a textual-based knowledge graph, 
 each entity of the first set of entities being represented by entity description text to describe a concept and meaning of the entities and each relation of the first set of relations being represented by relation description text to describe each relation between two or more entities; and 
 generating the second knowledge graph as an acoustic-based knowledge graph, each entity of the second set of entities being represented by entity description audio data to describe a concept and meaning of the entities and each relation of the second set of relations being represented by relation description audio data to describe each relation between two or more entities. 
 
     
     
       8. The method of  claim 1 , further comprising:
 pre-training the language module on a training data set comprising unlabeled text data to understand semantic information from text-based transcripts. 
 
     
     
       9. The method of  claim 1 , further comprising:
 pre-training a speech module on a training data set comprising unannotated acoustic data to understand semantic information from audio-based speech utterances. 
 
     
     
       10. The method of  claim 9 , further comprising:
 aligning the language module included in the integrated knowledge-language module with the pre-trained speech module; and 
 generating an optimized speech model configured to perform semantic analysis on audio data by leveraging acoustic information, language information and knowledge information. 
 
     
     
       11. A computer implemented method for joint training a speech module with a knowledge module for natural language understanding, the method comprising:
 obtaining a first acoustic knowledge graph comprising a first set of entities and a first set of relations, each relation of the first set of relations describing a relationship between two or more entities included in the first set of entities; 
 accessing a speech module comprising a first speech model and a second speech model, wherein the first speech model generates initial entity embeddings corresponding to the first acoustic knowledge graph and embeddings comprising contextual representations; 
 accessing and training a knowledge module to generate embeddings comprising final entity representations by applying the knowledge module to the initial entity embeddings; 
 training the speech module to generate context embeddings by applying the second speech model to the contextual representations and the final entity representations; 
 generating an integrated knowledge-speech module by integrating the speech module and the knowledge module by configuring the knowledge module to provide the final entity representations as knowledge information input to the speech module and configuring the speech module to provide context embeddings as acoustic information input to the knowledge module; 
 obtaining a second acoustic knowledge graph comprising a second set of entities and a second set of relations, each relation of the second set of relations describing a relationship between two or more entities included in the second set of entities; 
 aligning the first acoustic knowledge graph and second acoustic knowledge graph such that a first subset of entities and relations from the first acoustic knowledge graph corresponds to a second subset of entities and relations from the second acoustic knowledge graph; 
 further training the knowledge module by applying the aligned acoustic knowledge graphs to the knowledge module such that the integrated knowledge-speech module performs semantic analysis for the entities and entity relations in the second acoustic knowledge graph based on knowledge learned from entities and entity relations in the first acoustic knowledge graph. 
 
     
     
       12. The method of  claim 11 , further comprising:
 generating the first acoustic knowledge graph in a first language; 
 generating the second acoustic knowledge graph in a second language; 
 configuring the knowledge module to generate acoustic knowledge-based entity representations in the second language based on the first acoustic knowledge graph; and 
 training the integrated knowledge-speech module to perform semantic analysis in the second language based on knowledge learned from entities and entity relations in the first language. 
 
     
     
       13. The method of  claim 12 , further comprising:
 training the knowledge module to generate acoustic knowledge-based entity representations in the first language based on the second acoustic knowledge graph; and 
 training the integrated knowledge-speech module to perform semantic analysis in the first language based on knowledge learned from entities and entity relations in the second language. 
 
     
     
       14. The method of  claim 12 , further comprising:
 obtaining electronic content comprising a first set of speech audio data in the first language; 
 applying the electronic content as input to the integrated knowledge-speech module; and 
 translating the first set of speech audio data into a second set of speech audio data in the second language using the integrated knowledge-speech module. 
 
     
     
       15. A computing system configured for joint training a language module with a knowledge module for natural language understanding, the computing system comprising:
 one or more processors; and 
 one or more hardware storage devices that store computer-executable instructions that are structured to be executed by the one or more processors to cause the computing system to at least:
 obtain electronic content comprising speech utterances; 
 obtain an integrated generate an integrated knowledge-language module that is configured to perform semantic analysis for entities and entity relations in a second knowledge graph based on knowledge learned from entities and entity relations in a first knowledge graph and as a result of integrating a language module comprising a first language model and second language model with a knowledge module such that the knowledge module is configured to provide knowledge information generated by the knowledge module based on a first set of embeddings generated by the first language model to the language module and the language module is configure to provide acoustic information generated by the second language model based on a second set of embeddings from the first language model and the knowledge information, wherein the knowledge module is further trained with an alignment of the first knowledge graph and second knowledge graph; 
 perform semantic analysis on the electronic content using the integrated knowledge-language module; and 
 output a translation of the electronic content from the integrated knowledge-language module. 
 
 
     
     
       16. The computing system of  claim 15 , the first knowledge graph and second knowledge graph comprising acoustic-based knowledge graphs, the electronic content comprising acoustic-based speech utterances. 
     
     
       17. The computing system of  claim 15 , the first knowledge graph and second knowledge graph comprising language or textual-based knowledge graphs, the electronic content comprising text-based speech utterances. 
     
     
       18. The computing system of  claim 17 , wherein outputting a translation of the electronic content comprises translating a first set of speech transcriptions in a first language into a second set of speech transcriptions in a second language using the integrated knowledge-language module. 
     
     
       19. The computing system of  claim 15 , the computer-executable instructions being executable by the one or more processors to further cause the computing system to:
 perform semantic analysis on the electronic content in a second language based on knowledge learned from entities and entity relations in a first language. 
 
     
     
       20. The computing system of  claim 15 , the language module being aligned with a speech module, the speech module being pre-trained on unlabeled acoustic data to understand semantic information from speech utterances. 
     
     
       21. The method of  claim 1 , wherein the method further comprises:
 training the knowledge module includes applying the knowledge module to the first set of embeddings generated by the first language model; 
 generating the third set of embeddings as the knowledge information input from the knowledge module based on applying the knowledge module to the first set of embeddings from the first language model; and 
 training the language module by training the second language model to generate a fourth set of embeddings comprising context embeddings by applying the second language model to the second set of embeddings from the first language model and the third set of embeddings from the knowledge module.