Language model with external knowledge base

Abstract

The technology described herein receives a natural-language sequence of words comprising multiple entities. The technology then identifies a plurality of entities in the natural-language sequence. The technology generates a masked natural-language sequence by masking a first entity in the natural-language sequence. The technology retrieves, from a knowledge base, information related to a second entity in the plurality of entities. The technology then trains a natural-language model to respond to a query. The training uses a first representation of the masked natural-language sequence, a second representation of the information, and the first entity.

What is claimed is: 
     
       1. A computer-implemented method comprising:
 receiving a natural-language sequence of words comprising multiple entities, wherein the entities comprise persons and places; 
 identifying a plurality of entities in the natural-language sequence; 
 generating a masked natural-language sequence by masking a first entity in the natural-language sequence; 
 generating a first representation of the masked natural-language sequence, wherein the first representation is a machine embedding of the masked natural-language sequence; 
 retrieving, from a knowledge base, information related to a second entity in the plurality of entities; 
 generating a second representation of the information; and 
 training a natural-language model to respond to a query, wherein the training uses the first representation of the masked natural-language sequence and the second representation of the information as inputs and the first entity as a training label. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein the information is a triple comprising the second entity, a third entity, and a relationship between the second entity and the third entity. 
     
     
       3. The computer-implemented method of  claim 2 , further comprising generating a natural language phrase that includes the second entity, the third entity and the relationship. 
     
     
       4. The computer-implemented method of  claim 3 , wherein the second representation of the information is a machine embedding of the natural language phrase. 
     
     
       5. The computer-implemented method of  claim 2 , further comprising generating a similarity score between the first representation of the natural-language sequence of words and the second representation of the information. 
     
     
       6. The computer-implemented method of  claim 5 , wherein the similarity score is based on a relational similarity score between a machine embedding of the relationship from the triple and a machine embedding of the natural-language sequence of words. 
     
     
       7. The computer-implemented method of  claim 5 , wherein the information is associated with the similarity score above a threshold rank when stack ranked with similarity scores calculated for other information retrieved from the knowledge base. 
     
     
       8. The computer-implemented method of  claim 1 , wherein the training comprises masked language modelling where a training objective is to predict the first entity. 
     
     
       9. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by a processing device, cause the processing device to:
 receiving a query comprising a natural-language sequence of words; 
 identifying a plurality of entities in the natural-language sequence of words, wherein the entities comprise persons and places; 
 retrieving, from a knowledge base, information related to an entity in the plurality of entities, wherein the information is a triple comprising the entity, a second entity, and a relationship between the entity and the second entity; 
 generating a natural language phrase that includes the entity, the second entity and the relationship; 
 generating a first representation of the natural-language sequence comprising a machine embedding of the natural-language sequence; 
 generating a second representation of the information comprising a machine embedding of the natural language phrase; 
 providing the first representation of the natural-language sequence of words and the second representation of the information to a natural-language model; 
 in response to the providing, generating, using the natural-language model, a natural language response to the query; and 
 communicating the natural language response. 
 
     
     
       10. The non-transitory computer-readable medium of  claim 9 , wherein the natural language response is provided by a chat bot. 
     
     
       11. The non-transitory computer-readable medium of  claim 9 , wherein the response includes a natural language phrase. 
     
     
       12. The non-transitory computer-readable medium of  claim 9 , further comprising generating a similarity score between the first representation of the natural-language sequence of words and the second representation of the information. 
     
     
       13. The non-transitory computer-readable medium of  claim 12 , wherein the similarity score is based on a relational similarity score between a machine embedding of the relationship from the triple and the machine embedding of the natural-language sequence of words. 
     
     
       14. The non-transitory computer-readable medium of  claim 12 , wherein the information is associated with the similarity score above a threshold rank when stack ranked with similarity scores calculated for other information retrieved from the knowledge base. 
     
     
       15. A system comprising:
 a memory component; and 
 a processing device coupled to the memory component, the processing device to perform operations comprising: 
 receiving a natural-language sequence of words; 
 identifying an entity in the natural-language sequence of words; 
 retrieving, from a knowledge base, a plurality of triples related to the entity, wherein each triple in plurality of triples comprises the entity, a second entity, and a relationship between the entity and the second entity; 
 for each triple in the plurality of triples, calculating a similarity score that represents an amount of similarity between the natural-language sequence of words and an individual triple; 
 selecting a top plurality of triples from the plurality of triples using the similarity score; 
 providing a first representation of the natural-language sequence of words and a second representation of the top plurality of triples to a natural-language model; 
 in response to the providing, generating, using the natural-language model, a natural language response to the natural-language sequence of words; and 
 communicating the natural language response to a user. 
 
     
     
       16. The system of  claim 15 , wherein each of the plurality of triples comprise the entity, a second entity, and a relationship between the entity and the second entity. 
     
     
       17. The system of  claim 16 , further comprising generating a natural language phrase that includes the entity, the second entity and the relationship. 
     
     
       18. The system of  claim 17 , wherein the first representation of the natural-language sequence is a machine embedding of the natural-language sequence and the second representation of the top plurality of triples includes a machine embedding of the natural language phrase. 
     
     
       19. The system of  claim 18 , wherein the similarity score is based on a relational similarity score between a machine embedding of the relationship from the individual triple and a machine embedding of the natural-language sequence of words. 
     
     
       20. The system of  claim 18 , wherein the natural language response is provided by a chat bot.