Annotating and modeling natural language semantics through annotation conversion

Abstract

A natural language understanding (NLU) system generates in-place annotations for natural language utterances or other types of time-based media based on stand-off annotations. The in-place annotations are associated with particular sub-sequences of an annotation, which provides richer information than stand-off annotations, which are associated only with an utterance as a whole. To generate the in-place annotations for an utterance, the NLU system applies an encoder network and a decoder network to obtain attention weights for the various tokens within the utterance. The NLU system disqualifies tokens of the utterance based on their corresponding attention weights, and selects highest-scoring contiguous sequences of tokens between the disqualified tokens. In-place annotations are associated with the selected sequences.

What is claimed is: 
     
       1. A computer-implemented method for generating in-place annotations for an utterance, the method comprising:
 training a stand-off annotation model using utterances labeled with stand-off annotations, the training maximizing a likelihood of the label for an input utterance, the stand-off annotation model comprising an encoder network and a decoder network; 
 generating stand-off labels for the utterance, the generating comprising:
 generating, using the encoder network, encoder hidden states for text of the utterance, and 
 computing, for tokens of the text of the utterance using the decoder network, attention weights based at least in part on the encoder hidden states; 
 
 disqualifying tokens of text of the utterance having attention weights below a threshold attention value; 
 identifying contiguous sequences of tokens between disqualified tokens; 
 determining a score for each contiguous sequence of tokens based on an aggregate attention weight for tokens of the contiguous sequence scaled by a length of the contiguous sequence; 
 selecting one or more of the contiguous sequences having highest scores; and 
 generating in-place annotations for the selected one or more contiguous sequences. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein computing attention weights comprises computing an attention weight for <token, label> pair, where the tokens are the tokens of the text of the utterances, and the labels are from a predefined set of labels for a domain. 
     
     
       3. The computer-implemented method of  claim 1 , wherein disqualifying the tokens comprises determining whether the attention weights are below an attention value resulting from all tokens having a same attention value. 
     
     
       4. The computer-implemented method of  claim 1 , further comprising computing scores of the identified contiguous sequences by summing attention weights of tokens of the sequences and scaling the sum by a length of the sequence. 
     
     
       5. The computer-implemented method of  claim 1 , wherein the in-place annotations correspond to labels drawn from a predefined set of labels for a domain. 
     
     
       6. The computer-implemented method of  claim 5 , further comprising training an additional NLU model for a first one of the labels using one or more of the in-place annotations that correspond to the first one of the labels. 
     
     
       7. The computer-implemented method of  claim 6 , wherein the training comprises using equivalence classes of a domain ontology to generate additional training data by substituting equivalent terms for those of the in-place annotations. 
     
     
       8. A non-transitory computer-readable storage medium storing executable computer program instructions that when executed by a computer processor perform actions comprising:
 training a stand-off annotation model using utterances labeled with stand-off annotations, the training maximizing a likelihood of the label for an input utterance, the stand-off annotation model comprising an encoder network and a decoder network; 
 generating, using the encoder network, encoder hidden states for text of an utterance; 
 computing, for tokens of text of the utterance using the decoder network, attention weights based at least in part on the encoder hidden states; 
 disqualifying tokens of text of the utterance having attention weights below a threshold attention value; 
 identifying contiguous sequences of tokens between disqualified tokens; 
 determining a score for each contiguous sequence of tokens based on an aggregate attention weight for tokens of the contiguous sequence scaled by a length of the contiguous sequence; 
 selecting one or more of the contiguous sequences having highest scores; and 
 generating in-place annotations for the selected one or more contiguous sequences. 
 
     
     
       9. The non-transitory computer-readable storage medium of  claim 8 , wherein computing attention weights comprises computing an attention weight for <token, label> pair, where the tokens are the tokens of the text of the utterances, and the labels are from a predefined set of labels for a domain. 
     
     
       10. The non-transitory computer-readable storage medium of  claim 8 , wherein disqualifying the tokens comprises determining whether the attention weights are below an attention value resulting from all tokens having a same attention value. 
     
     
       11. The non-transitory computer-readable storage medium of  claim 8 , the actions further comprising computing scores of the identified contiguous sequences by summing attention weights of tokens of the sequences and scaling the sum by a length of the sequence. 
     
     
       12. The non-transitory computer-readable storage medium of  claim 8 , wherein the in-place annotations correspond to labels drawn from a predefined set of labels for a domain. 
     
     
       13. The non-transitory computer-readable storage medium of  claim 12 , the actions further comprising training an additional NLU model for a first one of the labels using one or more of the in-place annotations that correspond to the first one of the labels. 
     
     
       14. The non-transitory computer-readable storage medium of  claim 13 , wherein the training comprises using equivalence classes of a domain ontology to generate additional training data by substituting equivalent terms for those of the in-place annotations. 
     
     
       15. A natural language understanding (NLU) system comprising:
 a computer processor; and 
 a non-transitory computer-readable storage medium storing instructions that when executed by the computer processor perform actions comprising:
 training a stand-off annotation model using utterances labeled with stand-off annotations, the training maximizing a likelihood of the label for an input utterance, the stand-off annotation model comprising an encoder network and a decoder network;
 generating, using the encoder network, encoder hidden states for text of the utterance; 
 computing, for tokens of text of the utterance using the decoder network, attention weights based at least in part on the encoder hidden states; 
 disqualifying tokens of text of the utterance having attention weights below a threshold attention value; 
 identifying contiguous sequences of tokens between disqualified tokens; 
 determining a score for each contiguous sequence of tokens based on an aggregate attention weight for tokens of the contiguous sequence scaled by a length of the contiguous sequence; 
 selecting one or more of the contiguous sequences having highest scores; and 
 generating in-place annotations for the one or more selected contiguous sequences. 
 
 
 
     
     
       16. The NLU system of  claim 15 , wherein computing attention weights comprises computing an attention weight for <token, label> pair, where the tokens are the tokens of the text of the utterances, and the labels are from a predefined set of labels for a domain. 
     
     
       17. The NLU system of  claim 15 , wherein disqualifying the tokens comprises determining whether the attention weights are below an attention value resulting from all tokens having a same attention value. 
     
     
       18. The NLU system of  claim 15 , the actions further comprising computing scores of the identified contiguous sequences by summing attention weights of tokens of the sequences and scaling the sum by a length of the sequence. 
     
     
       19. The NLU system of  claim 15 , wherein the in-place annotations correspond to labels drawn from a predefined set of labels for a domain. 
     
     
       20. A computer-implemented method for generating in-place annotations for time-based media comprising a sequence of temporal units, the method comprising:
 training a stand-off annotation model using utterances labeled with stand-off annotations, the training maximizing a likelihood of the label for an input utterance, the stand-off annotation model comprising an encoder network and a decoder network; 
 generating, using the encoder network, encoder hidden states for temporal units of the media; 
 computing, for temporal units of the media using the decoder network, attention weights based at least in part on the encoder hidden states; 
 disqualifying temporal units of the media having attention weights below a threshold attention value; 
 identifying contiguous sequences of temporal units between disqualified temporal units; 
 determining a score for each contiguous sequence of temporal units based on an aggregate attention weight for temporal units of the contiguous sequence scaled by a length of the contiguous sequence; 
 selecting one or more of the contiguous sequences having highest scores; and 
 generating in-place annotations for the selected one or more contiguous sequences.