Alignment prediction to inject text into automatic speech recognition training

Abstract

A method includes receiving training data that includes unspoken textual utterances, un-transcribed non-synthetic speech utterances, and transcribed non-synthetic speech utterances. Each unspoken textual utterance is not paired with any corresponding spoken utterance of non-synthetic speech. Each un-transcribed non-synthetic speech utterance not paired with a corresponding transcription. Each transcribed non-synthetic speech utterance paired with a corresponding transcription. The method also includes generating a corresponding alignment output for each unspoken textual utterance of the received training data using an alignment model. The method also includes pre-training an audio encoder on the alignment outputs generated for corresponding to the unspoken textual utterances, the un-transcribed non-synthetic speech utterances, and the transcribed non-synthetic speech utterances to teach the audio encoder to jointly learn shared speech and text representations.

What is claimed is: 
     
         1 . A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
 receiving training data comprising:
 unspoken textual utterances, each unspoken textual utterance not paired with any corresponding spoken utterance of non-synthetic speech; 
 un-transcribed non-synthetic speech utterances, each un-transcribed non-synthetic speech utterance not paired with a corresponding transcription; and 
 transcribed non-synthetic speech utterances, each transcribed non-synthetic speech utterance paired with a corresponding transcription; 
   for each respective unspoken textual utterance of the received training data, generating, using an alignment model, a corresponding alignment output that maps the respective unspoken textual utterance to a corresponding predicted speech duration of the respective unspoken textual utterance; and   pre-training an audio encoder on the corresponding alignment output generated for each respective unspoken textual utterance, the un-transcribed non-synthetic speech utterances, and the transcribed non-synthetic speech utterances to teach the audio encoder to jointly learn shared speech and text representations by updating parameters of the audio encoder, the audio encoder comprising a stack of self-attention layers each including a multi-headed self-attention mechanism,   wherein an automatic speech recognition (ASR) model comprising the pre-trained audio encoder is configured to receive a spoken utterance and generate a corresponding transcription of the spoken utterance.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein pre-training the audio encoder comprises:
 for each un-transcribed non-synthetic speech utterance:
 generating a corresponding encoded representation of the un-transcribed non-synthetic speech utterance; and 
 pre-training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the un-transcribed non-synthetic speech utterance; 
   for each alignment output:
 generating a corresponding encoded representation of the alignment output; and 
 pre-training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the alignment output; and 
   for each transcribed non-synthetic speech utterance:
 generating a corresponding encoded representation of the transcribed non-synthetic speech utterance; and 
 pre-training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the transcribed non-synthetic speech utterance. 
   
     
     
         3 . The computer-implemented method of  claim 1 , wherein pre-training the audio encoder comprises:
 at each of a plurality of time steps for each alignment output:
 generating, using an auxiliary decoder, a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding alignment output; 
 determining an alignment output loss term based on the first probability distribution over possible synthetic speech recognition hypotheses and the unspoken textual utterance corresponding to the alignment output; and 
 pre-training the audio encoder based on the alignment output loss term; and 
   at each of a plurality of time steps for each transcribed non-synthetic speech utterance:
 generating, using the auxiliary decoder, a second probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance; 
 determining a non-synthetic speech loss term based on the second probability distribution over possible non-synthetic speech recognition hypotheses and the corresponding transcription paired with the transcribed non-synthetic speech utterance; and 
 pre-training the audio encoder based on the non-synthetic speech loss term. 
   
     
     
         4 . The computer-implemented method of  claim 3 , wherein the auxiliary decoder comprises one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or Recurrent Neural Network-Transducer (RNN-T) decoder. 
     
     
         5 . The computer-implemented method of  claim 3 , wherein:
 the first probability distribution over possible synthetic speech recognition hypotheses comprises one of possible phoneme labels or possible word piece labels; and   the second probability distribution over possible non-synthetic speech recognition hypotheses comprises the one of the possible phoneme labels or the possible word piece labels.   
     
     
         6 . The computer-implemented method of  claim 1 , wherein the audio encoder comprises a text encoder, a speech encoder, and a shared encoder. 
     
     
         7 . The computer-implemented method of  claim 6 , wherein the operations further comprise:
 for each alignment output:
 determining, using the text encoder, an encoded textual representation of the alignment output; and 
 generating, using the shared encoder, a first encoded shared representation of the alignment output in a shared latent representation space; and 
   for each transcribed non-synthetic speech utterance:
 determining, using the speech encoder, an encoded audio representation of the transcribed non-synthetic speech utterance; and 
 generating, using the shared encoder, a second encoded shared representation of the transcribed non-synthetic speech utterance in a shared latent representation space. 
   
     
     
         8 . The computer-implemented method of  claim 1 , wherein generating the corresponding alignment output for each unspoken textual utterance of the received training data comprises:
 extracting an initial textual representation from the unspoken textual utterance;   predicting a text chunk duration for each text chunk in the unspoken textual utterance; and   upsampling the initial textual representation using the predicted text chunk duration for each text chunk in the unspoken textual utterance.   
     
     
         9 . The computer-implemented method of  claim 1 , wherein the operations further comprise training the alignment model by:
 generating, using a speech encoder, an encoded audio representation for a transcribed non-synthetic speech utterance;   determining, using the alignment model, an alignment output for a transcription corresponding to the transcribed non-synthetic speech utterance;   generating, using a text encoder, an encoded textual representation for the alignment output; and   updating parameters of the alignment model based on a comparison of the encoded audio representation for the transcribed non-synthetic speech utterance and encoded textual representation for the alignment output.   
     
     
         10 . A system comprising:
 data processing hardware; and   memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
 receiving training data comprising:
 unspoken textual utterances, each unspoken textual utterance not paired with any corresponding spoken utterance of non-synthetic speech; 
 un-transcribed non-synthetic speech utterances, each un-transcribed non-synthetic speech utterance not paired with a corresponding transcription; and 
 transcribed non-synthetic speech utterances, each transcribed non-synthetic speech utterance paired with a corresponding transcription; 
 
 for each respective unspoken textual utterance of the received training data, generating, using an alignment model, a corresponding alignment output that maps the respective unspoken textual utterance to a corresponding predicted speech duration of the respective unspoken textual utterance; and 
 pre-training an audio encoder on the corresponding alignment output generated for each respective unspoken textual utterance, the un-transcribed non-synthetic speech utterances, and the transcribed non-synthetic speech utterances to teach the audio encoder to jointly learn shared speech and text representations by updating parameters of the audio encoder, the audio encoder comprising a stack of self-attention layers each including a multi-headed self-attention mechanism, 
 wherein an automatic speech recognition (ASR) model comprising the pre-trained audio encoder is configured to receive a spoken utterance and generate a corresponding transcription of the spoken utterance. 
   
     
     
         11 . The system of  claim 10 , wherein pre-training the audio encoder comprises:
 for each un-transcribed non-synthetic speech utterance:
 generating a corresponding encoded representation of the un-transcribed non-synthetic speech utterance; and 
 pre-training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the un-transcribed non-synthetic speech utterance; 
   for each alignment output:
 generating a corresponding encoded representation of the alignment output; and 
 pre-training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the alignment output; and 
   for each transcribed non-synthetic speech utterance:
 generating a corresponding encoded representation of the transcribed non-synthetic speech utterance; and 
 pre-training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the transcribed non-synthetic speech utterance. 
   
     
     
         12 . The system of  claim 10 , wherein pre-training the audio encoder comprises:
 at each of a plurality of time steps for each alignment output:
 generating, using an auxiliary decoder, a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding alignment output; 
 determining an alignment output loss term based on the first probability distribution over possible synthetic speech recognition hypotheses and the unspoken textual utterance corresponding to the alignment output; and 
 pre-training the audio encoder based on the alignment output loss term; and 
   at each of a plurality of time steps for each transcribed non-synthetic speech utterance:
 generating, using the auxiliary decoder, a second probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance; 
 determining a non-synthetic speech loss term based on the second probability distribution over possible non-synthetic speech recognition hypotheses and the corresponding transcription paired with the transcribed non-synthetic speech utterance; and 
 pre-training the audio encoder based on the non-synthetic speech loss term. 
   
     
     
         13 . The system of  claim 12 , wherein the auxiliary decoder comprises one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or Recurrent Neural Network-Transducer (RNN-T) decoder. 
     
     
         14 . The system of  claim 12 , wherein:
 the first probability distribution over possible synthetic speech recognition hypotheses comprises one of possible phoneme labels or possible word piece labels; and   the second probability distribution over possible non-synthetic speech recognition hypotheses comprises the one of the possible phoneme labels or the possible word piece labels.   
     
     
         15 . The system of  claim 10 , wherein the audio encoder comprises a text encoder, a speech encoder, and a shared encoder. 
     
     
         16 . The system of  claim 15 , wherein the operations further comprise:
 for each alignment output:
 determining, using the text encoder, an encoded textual representation of the alignment output; and 
 generating, using the shared encoder, a first encoded shared representation of the alignment output in a shared latent representation space; and 
   for each transcribed non-synthetic speech utterance:
 determining, using the speech encoder, an encoded audio representation of the transcribed non-synthetic speech utterance; and 
 generating, using the shared encoder, a second encoded shared representation of the transcribed non-synthetic speech utterance in a shared latent representation space. 
   
     
     
         17 . The system of  claim 10 , wherein generating the corresponding alignment output for each unspoken textual utterance of the received training data comprises:
 extracting an initial textual representation from the unspoken textual utterance;   predicting a text chunk duration for each text chunk in the unspoken textual utterance; and   upsampling the initial textual representation using the predicted text chunk duration for each text chunk in the unspoken textual utterance.   
     
     
         18 . The system of  claim 10 , wherein the operations further comprise training the alignment model by:
 generating, using a speech encoder, an encoded audio representation for a transcribed non-synthetic speech utterance;   determining, using the alignment model, an alignment output for a transcription corresponding to the transcribed non-synthetic speech utterance;   generating, using a text encoder, an encoded textual representation for the alignment output; and   updating parameters of the alignment model based on a comparison of the encoded audio representation for the transcribed non-synthetic speech utterance and encoded textual representation for the alignment output.