Natural language text generation from a set of keywords using machine learning and templates

Abstract

An embodiment of the present invention generates natural language content from a set of keywords in accordance with a template. Keyword vectors representing a context for the keywords are generated. The keywords are associated with language tags, while the template includes a series of language tags indicating an arrangement for the generated natural language content. Template vectors are generated from the series of language tags of the template and represent a context for the template. Contributions from the contexts for the keywords and the template are determined based on a comparison of the series of language tags of the template with the associated language tags of the keywords. One or more words for each language tag of the template are generated to produce the natural language content based on combined contributions from the contexts for the keywords and the template.

What is claimed is: 
     
       1. A method of generating natural language content from a set of keywords in accordance with a template comprising:
 generating, via a processor, word embeddings for the keywords; 
 generating, via the processor, keyword vectors representing a context for the keywords based on the word embeddings for the keywords, wherein the keywords are syntactically unordered and associated with language tags, and wherein the template includes a series of language tags indicating an arrangement for words of the generated natural language content; 
 generating, via the processor, word embeddings for the series of language tags of the template; 
 generating, via the processor, template vectors based on the word embeddings for the series of language tags of the template, wherein the template vectors represent a context for the template; 
 generating, via the processor, word embeddings for the associated language tags of the keywords; 
 determining, via the processor, contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors based on a comparison of the word embeddings of the series of language tags of the template with the word embeddings of the associated language tags of the keywords; and 
 generating, via a machine learning model of the processor, one or more words for each language tag of the template from a word vocabulary to produce the natural language content based on combined contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors, wherein the machine learning model includes a recurrent neural network and the word vocabulary is learned from training data during training of the machine learning model. 
 
     
     
       2. The method of  claim 1 , wherein the language tags of the template and the associated language tags of the keywords include part-of-speech tags. 
     
     
       3. The method of  claim 1 , wherein determining contributions comprises:
 determining a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords, wherein the probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and wherein a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template. 
 
     
     
       4. The method of  claim 3 , further comprising:
 applying the probability for the corresponding language tag of the template to a keyword vector associated with the corresponding language tag of the template to produce the contribution of the context for the keywords; 
 applying the complement of the probability for the corresponding language tag of the template to a template vector associated with the corresponding language tag of the template to produce the contribution of the context for the template; and 
 combining the contributions of the contexts for the keywords and the template to produce the combined contributions. 
 
     
     
       5. The method of  claim 1 , further comprising:
 determining the associated language tags for the keywords via a second machine learning model, wherein the second machine learning model is trained with a data set including complete sentences and the complete sentences without function words. 
 
     
     
       6. The method of  claim 1 , wherein the keywords are in a first natural language, and the generated natural language content is in a second different natural language. 
     
     
       7. The method of  claim 1 , wherein generating the keyword vectors comprises:
 encoding the word embeddings for the keywords using a second machine learning model to produce encoded vector representations of the keywords, wherein the second machine learning model is trained to produce the same encoded vector representations for a corresponding set of keywords regardless of an order of keywords in the corresponding set; and 
 generating the keyword vectors based on the encoded vector representations. 
 
     
     
       8. The method of  claim 7 , wherein generating the keyword vectors based on the encoded vector representations further comprises:
 applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations, wherein the attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template. 
 
     
     
       9. The method of  claim 1 , wherein generating the template vectors comprises:
 encoding the word embeddings for the series of language tags of the template using a bidirectional recurrent machine learning model; and 
 producing the template vectors based on the encoded word embeddings for the series of language tags of the template, wherein each template vector is produced based on adjacent language tags within the template. 
 
     
     
       10. The method of  claim 1 , wherein generating one or more words for each language tag of the template comprises:
 determining for each language tag of the template a probability distribution over the word vocabulary using the machine learning model; and 
 selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution. 
 
     
     
       11. A system for generating natural language content from a set of keywords in accordance with a template comprising:
 a processor configured to:
 generate word embeddings for the keywords; 
 generate keyword vectors representing a context for the keywords based on the word embeddings for the keywords, wherein the keywords are syntactically unordered and associated with language tags, and wherein the template includes a series of language tags indicating an arrangement for words of the generated natural language content; 
 generate word embeddings for the series of language tags of the template; 
 generate template vectors based on the word embeddings for the series of language tags of the template, wherein the template vectors represent a context for the template; 
 generate word embeddings for the associated language tags of the keywords; 
 determine contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors based on a comparison of the word embeddings of the series of language tags of the template with the word embeddings of the associated language tags of the keywords; and 
 generate, via a machine learning model, one or more words for each language tag of the template from a word vocabulary to produce the natural language content based on combined contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors, wherein the machine learning model includes a recurrent neural network and the word vocabulary is learned from training data during training of the machine learning model. 
 
 
     
     
       12. The system of  claim 11 , wherein determining contributions comprises:
 determining a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords, wherein the probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and wherein a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template. 
 
     
     
       13. The system of  claim 11 , wherein the processor is further configured to:
 determine the associated language tags for the keywords via a second machine learning model, wherein the second machine learning model is trained with a data set including complete sentences and the complete sentences without function words. 
 
     
     
       14. The system of  claim 11 , wherein generating the keyword vectors comprises:
 encoding the word embeddings for the keywords using a second machine learning model to produce encoded vector representations of the keywords, wherein the second machine learning model is trained to produce the same encoded vector representations for a corresponding set of keywords regardless of an order of keywords in the corresponding set; and 
 generating the keyword vectors based on the encoded vector representations, wherein generating the keyword vectors based on the encoded vector representations further comprises:
 applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations, wherein the attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template; and 
 
 wherein generating the template vectors comprises:
 encoding the word embeddings for the series of language tags of the template using a bidirectional recurrent machine learning model; and 
 producing the template vectors based on the encoded word embeddings for the series of language tags of the template, wherein each template vector is produced based on adjacent language tags within the template. 
 
 
     
     
       15. The system of  claim 11 , wherein generating one or more words for each language tag of the template comprises:
 determining for each language tag of the template a probability distribution over the word vocabulary using the machine learning model; and 
 selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution. 
 
     
     
       16. A computer program product for generating natural language content from a set of keywords in accordance with a template, the computer program product comprising one or more computer readable storage media having program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by a processor to cause the processor to:
 generate word embeddings for the keywords; 
 generate keyword vectors representing a context for the keywords based on the word embeddings for the keywords, wherein the keywords are syntactically unordered and associated with language tags, and wherein the template includes a series of language tags indicating an arrangement for words of the generated natural language content; 
 generate word embeddings for the series of language tags of the template; 
 generate template vectors based on the word embeddings for the series of language tags of the template, wherein the template vectors represent a context for the template; 
 generate word embeddings for the associated language tags of the keywords; 
 determine contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors based on a comparison of the word embeddings of the series of language tags of the template with the word embeddings of the associated language tags of the keywords; and 
 generate, via a machine learning model, one or more words for each language tag of the template from a word vocabulary to produce the natural language content based on combined contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors, wherein the machine learning model includes a recurrent neural network and the word vocabulary is learned from training data during training of the machine learning model. 
 
     
     
       17. The computer program product of  claim 16 , wherein determining contributions comprises:
 determining a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords, wherein the probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and wherein a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template. 
 
     
     
       18. The computer program product of  claim 16 , wherein the program instructions further cause the processor to:
 determine the associated language tags for the keywords via a second machine learning model, wherein the second machine learning model is trained with a data set including complete sentences and the complete sentences without function words. 
 
     
     
       19. The computer program product of  claim 16 , wherein generating the keyword vectors comprises:
 encoding the word embeddings for the keywords using a second machine learning model to produce encoded vector representations of the keywords, wherein the second machine learning model is trained to produce the same encoded vector representations for a corresponding set of keywords regardless of an order of keywords in the corresponding set; and 
 generating the keyword vectors based on the encoded vector representations, wherein generating the keyword vectors based on the encoded vector representations further comprises:
 applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations, wherein the attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template; and 
 
 wherein generating the template vectors comprises:
 encoding the word embeddings for the series of language tags of the template using a bidirectional recurrent machine learning model; and 
 producing the template vectors based on the encoded word embeddings for the series of language tags of the template, wherein each template vector is produced based on adjacent language tags within the template. 
 
 
     
     
       20. The computer program product of  claim 16 , wherein generating one or more words for each language tag of the template comprises:
 determining for each language tag of the template a probability distribution over the word vocabulary using the machine learning model; and 
 selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution.