Word lattice augmentation for automatic speech recognition

Abstract

Speech processing techniques are disclosed that enable determining a text representation of named entities in captured audio data. Various implementations include determining the location of a carrier phrase in a word lattice representation of the captured audio data, where the carrier phrase provides an indication of a named entity. Additional or alternative implementations include matching a candidate named entity with the portion of the word lattice, and augmenting the word lattice with the matched candidate named entity.

What is claimed is: 
     
       1. A method implemented by one or more processors, the method comprising:
 generating a text representation of captured audio data using an automatic speech recognition engine, wherein generating the text representation of the captured audio data comprises:
 generating a word lattice corresponding to the captured audio data, wherein the word lattice is an acyclic graph representing candidate recognitions of the captured audio data; 
 determining, based on a location of a carrier phrase in the word lattice, an augmentation location in the word lattice; 
 selecting, from a set of candidate contextually relevant named entities, a contextually relevant named entity to insert at the augmentation location of the word lattice,
 wherein the contextually relevant named entity is an additional candidate recognition of a portion of the captured audio data, and 
 wherein selecting, from the set of candidate contextually relevant named entities, the contextually relevant named entity to insert at the augmentation location of the word lattice comprises:
 determining a set of candidate phoneme sequences corresponding to the word lattice at the augmentation location, wherein determining the set of candidate phoneme sequences corresponding to the word lattice at the augmentation location comprises: 
  for each phoneme in the set of candidate phoneme sequences: 
  determining whether the phoneme has one or more acoustically similar phonemes; and 
  in response to determining the phoneme has one or more acoustically similar phonemes, inserting the one or more acoustically similar phonemes as one or more additional sequence paths in the set of candidate phoneme sequences; and 
 selecting the contextually relevant named entity by matching a phoneme representation of a candidate contextually relevant named entity in the set of candidate contextually relevant named entities with a candidate phoneme sequence in the set of candidate phoneme sequences; 
 
 
 augmenting the word lattice by inserting the selected contextually relevant named entity at the augmentation location; and 
 processing the augmented word lattice to determine the text representation of the captured audio data. 
 
 
     
     
       2. The method of  claim 1 , wherein determining, based on the location of the carrier phrase in the word lattice, the augmentation location in the word lattice comprises processing the word lattice using a semantic pattern finite state transducer to determine the location of the carrier phrase in the word lattice. 
     
     
       3. The method of  claim 1 , wherein determining the set of candidate phoneme sequences corresponding to the word lattice at the augmentation location further comprises:
 determining the set of candidate phoneme sequences corresponding to the word lattice at the augmentation location without determining candidate phonemes for words in the word lattice preceding the augmentation location. 
 
     
     
       4. The method of  claim 1 , wherein generating the word lattice corresponding to the captured audio data comprises:
 generating the word lattice corresponding to the captured audio data using a first-pass decoding portion of the automatic speech recognition engine. 
 
     
     
       5. The method of  claim 1 , wherein processing the augmented word lattice to determine the text representation of the captured audio data comprises:
 processing the word lattice using a second-pass rescoring portion of the automated speech recognition engine; and 
 processing the augmented word lattice using a named entity engine. 
 
     
     
       6. The method of  claim 1 , wherein the contextually relevant named entity is a location, an event, or a person. 
     
     
       7. The method of  claim 1 , wherein the automatic speech recognition engine is trained using a set of training instances, and wherein the contextually relevant named entity is not in the set of training instances. 
     
     
       8. The method of  claim 1 , wherein the automatic speech recognition engine is trained using a set of training instances, and wherein the contextually relevant named entity occurs a number of times, in the set of training instances, that is below a threshold value. 
     
     
       9. The method of  claim 1 , further comprising identifying the set of candidate contextually relevant named entities based on the set of candidate contextually relevant named entities being stored in association with a user profile. 
     
     
       10. A computing device comprising:
 memory storing instructions; 
 one or more processors executing the instructions to:
 generate a text representation of captured audio data using an automatic speech recognition engine, wherein in generating the text representation of the captured audio data one or more of the processors are to: 
 generate a word lattice corresponding to the captured audio data, wherein the word lattice is an acyclic graph representing candidate recognitions of the captured audio data; 
 determine, based on a location of a carrier phrase in the word lattice, an augmentation location in the word lattice; 
 select, from a set of candidate contextually relevant named entities, a contextually relevant named entity to insert at the augmentation location of the word lattice, wherein the contextually relevant named entity is an additional candidate recognition of a portion of the captured audio data, and wherein selecting, from the set of candidate contextually relevant named entities, the contextually relevant named entity to insert at the augmentation location of the word lattice comprises:
 determining a set of candidate phoneme sequences corresponding to the word lattice at the augmentation location, wherein determining the set of candidate phoneme sequences corresponding to the word lattice at the augmentation location comprises: 
 for each candidate phoneme sequence in the set of candidate phoneme sequences:
 inserting one or more edited phoneme sequences as additional sequence paths in the set of candidate phoneme sequences, 
 where each edited phoneme sequence is a defined distance away from the candidate phoneme sequence; and 
 
 selecting the contextually relevant named entity by matching a phoneme representation of a candidate contextually relevant named entity in the set of candidate contextually relevant named entities with a candidate phoneme sequence in the set of candidate phoneme sequences; 
 
 augment the word lattice by inserting the selected contextually relevant named entity at the augmentation location; and 
 process the augmented word lattice to determine the text representation of the captured audio data. 
 
 
     
     
       11. The computing device of  claim 10 , wherein the computing device is a client device, and further comprising:
 one or more microphones, wherein the captured audio data is captured by the one or more microphones. 
 
     
     
       12. The computing device of  claim 10 , wherein in determining, based on the location of the carrier phrase in the word lattice, the augmentation location in the word lattice, one or more of the processors are to:
 process the word lattice using a semantic pattern finite state transducer to determine the location of the carrier phrase in the word lattice. 
 
     
     
       13. The computing device of  claim 10 , wherein in selecting, from the set of candidate contextually relevant named entities, the contextually relevant named entity to insert at the augmentation location of the word lattice, one or more of the processors are to:
 determine a set of candidate phoneme sequences corresponding to the word lattice at the augmentation location; and 
 select the contextually relevant named entity by matching a phoneme representation of a candidate contextually relevant named entity in the set of candidate contextually relevant named entities with a candidate phoneme sequence in the set of candidate phoneme sequences. 
 
     
     
       14. The computing device of  claim 13 , wherein in determining the set of candidate phoneme sequences corresponding to the word lattice at the augmentation location, one or more of the processors are to:
 determine the set of candidate phoneme sequences corresponding to the word lattice at the augmentation location without determining candidate phonemes for words in the word lattice preceding the augmentation location.