Speech recognition using unspoken text and speech synthesis

Abstract

A method for training a generative adversarial network (GAN)-based text-to-speech (TTS) model and a speech recognition model in unison includes obtaining a plurality of training text utterances. At each of a plurality of output steps for each training text utterance, the method also includes generating, for output by the GAN-Based TTS model, a synthetic speech representation of the corresponding training text utterance, and determining, using an adversarial discriminator of the GAN, an adversarial loss term indicative of an amount of acoustic noise disparity in one of the non-synthetic speech representations selected from the set of spoken training utterances relative to the corresponding synthetic speech representation of the corresponding training text utterance. The method also includes updating parameters of the GAN-based TTS model based on the adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances.

What is claimed is: 
     
       1. A computer-implemented method for training a speech recognition model, the method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
 obtaining a plurality of unspoken text utterances associated with a target domain the speech recognition model is being trained to learn; 
 obtaining a set of spoken training utterances, each spoken training utterance comprising a corresponding transcription paired with a corresponding non-synthetic speech representation of the spoken training utterance; 
 for each unspoken text utterance:
 conditioning the unspoken text utterance on a randomly assigned speaker embedding from a set of speaker embeddings, each speaker embedding in the set of speaker embeddings representing speaker characteristics; and 
 generating, as output from a pre-trained text-to-speech (TTS) model configured to receive the corresponding unspoken text utterance as input, a synthetic speech representation of the corresponding unspoken text utterance conditioned on the randomly assigned speaker embedding; and 
 
 training the speech recognition model on the synthetic speech representations generated as output from the pre-trained TTS model and the non-synthetic speech representations in the set of spoken training utterances, the speech recognition model comprising frame alignment-based transducer model. 
 
     
     
       2. The method of  claim 1 , wherein each speaker embedding in the set of speaker embeddings is extracted from a corresponding one of the non-synthetic speech representations in the set of spoken training utterances and represents the speaker characteristics of a speaker that spoke the corresponding spoken training utterance. 
     
     
       3. The method of  claim 1 , wherein each unspoken text utterance is represented by a corresponding sequence of phonemes. 
     
     
       4. The method of  claim 1 , wherein each synthetic speech representation generated as output from the pre-trained TTS model is represented by a sequence of mel-frequency spectrogram frames. 
     
     
       5. The method of  claim 1 , wherein each non-synthetic speech representation in the set of spoken training utterances is represented by a sequence of mel-frequency spectrogram frames. 
     
     
       6. The method of  claim 1 , wherein the speech recognition model comprising the frame alignment-based transducer model comprises a Recurrent Neural Network-Transducer (RNN-T) model. 
     
     
       7. The method of  claim 1 , wherein the pre-trained TTS model comprises:
 an encoder neural network configured to:
 receive, as input, each unspoken text utterance as a sequence of phonemes; and 
 generate, as output, a sequence of context vectors; and 
 
 a decoder neural network configured to:
 receive, as input, each context vector in the sequence of context vectors generated as output by the encoder neural network; and 
 generate, as output for each context vector, a corresponding frame in a sequence of mel-frequency spectrogram frames. 
 
 
     
     
       8. The method of  claim 1 , wherein the operations further comprise, while training the speech recognition model:
 at each of a plurality of output steps for each synthetic speech representation generated as output from the pre-trained TTS model:
 determining, for output by the speech recognition model, a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation; and 
 generating, by the data processing hardware, a synthetic speech loss term based on the first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation and the corresponding unspoken text utterance from which the corresponding synthetic speech representation is generated; and 
 
 at each of a plurality of output steps for each non-synthetic speech representation:
 determining, for output by the speech recognition model, a second probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation; and 
 generating a non-synthetic speech loss term based on the second probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation and the transcription in the set of spoken training utterances that is paired with the corresponding non-synthetic speech representation. 
 
 
     
     
       9. The method of  claim 1 , wherein the operations further comprise conditioning each unspoken text utterance on an utterance embedding selected from a set of utterance embeddings, each utterance embedding in the set of utterance embeddings representing an intended prosody. 
     
     
       10. The method of  claim 9 , wherein each utterance embedding in the set of utterance embeddings is extracted from a corresponding one of the non-synthetic speech representations in the set of spoken training utterances by a variational autoencoder (VAE). 
     
     
       11. A system for training a speech recognition model, the system comprising:
 data processing hardware; and 
 memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
 obtaining a plurality of unspoken text utterances associated with a target domain the speech recognition model is being trained to learn; 
 obtaining a set of spoken training utterances, each spoken training utterance comprising a corresponding transcription paired with a corresponding non-synthetic speech representation of the spoken training utterance; 
 for each unspoken text utterance:
 conditioning the unspoken text utterance on a randomly assigned speaker embedding from a set of speaker embeddings, each speaker embedding in the set of speaker embeddings representing speaker characteristics; and 
 generating, as output from a pre-trained text-to-speech (TTS) model configured to receive the corresponding unspoken text utterance as input, a synthetic speech representation of the corresponding unspoken text utterance conditioned on the randomly assigned speaker embedding; and 
 
 training the speech recognition model on the synthetic speech representations generated as output from the pre-trained TTS model and the non-synthetic speech representations in the set of spoken training utterances, the speech recognition model comprising frame alignment-based transducer model. 
 
 
     
     
       12. The system of  claim 11 , wherein each speaker embedding in the set of speaker embeddings is extracted from a corresponding one of the non-synthetic speech representations in the set of spoken training utterances and represents the speaker characteristics of a speaker that spoke the corresponding spoken training utterance. 
     
     
       13. The system of  claim 11 , wherein each unspoken text utterance is represented by a corresponding sequence of phonemes. 
     
     
       14. The system of  claim 11 , wherein each synthetic speech representation generated as output from the pre-trained TTS model is represented by a sequence of mel-frequency spectrogram frames. 
     
     
       15. The system of  claim 11 , wherein each non-synthetic speech representation in the set of spoken training utterances is represented by a sequence of mel-frequency spectrogram frames. 
     
     
       16. The system of  claim 11 , wherein the speech recognition model comprising the frame alignment-based transducer model comprises a Recurrent Neural Network-Transducer (RNN-T) model. 
     
     
       17. The system of  claim 11 , wherein the pre-trained TTS model comprises:
 an encoder neural network configured to:
 receive, as input, each unspoken text utterance as a sequence of phonemes; and 
 generate, as output, a sequence of context vectors; and 
 
 a decoder neural network configured to:
 receive, as input, each context vector in the sequence of context vectors generated as output by the encoder neural network; and 
 generate, as output for each context vector, a corresponding frame in a sequence of mel-frequency spectrogram frames. 
 
 
     
     
       18. The system of  claim 11 , wherein the operations further comprise, while training the speech recognition model:
 at each of a plurality of output steps for each synthetic speech representation generated as output from the pre-trained TTS model:
 determining, for output by the speech recognition model, a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation; and 
 generating, by the data processing hardware, a synthetic speech loss term based on the first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation and the corresponding unspoken text utterance from which the corresponding synthetic speech representation is generated; and 
 
 at each of a plurality of output steps for each non-synthetic speech representation:
 determining, for output by the speech recognition model, a second probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation; and 
 generating a non-synthetic speech loss term based on the second probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation and the transcription in the set of spoken training utterances that is paired with the corresponding non-synthetic speech representation. 
 
 
     
     
       19. The system of  claim 11 , wherein the operations further comprise conditioning each unspoken text utterance on an utterance embedding selected from a set of utterance embeddings, each utterance embedding in the set of utterance embeddings representing an intended prosody. 
     
     
       20. The system of  claim 19 , wherein each utterance embedding in the set of utterance embeddings is extracted from a corresponding one of the non-synthetic speech representations in the set of spoken training utterances by a variational autoencoder (VAE).