Unsupervised alignment for text to speech synthesis using neural networks

Abstract

Generation of synthetic speech from an input text sequence may be difficult when durations of individual phonemes forming the input text sequence are unknown. A predominantly parallel process may model speech rhythm as a separate generative distribution such that phoneme duration may be sampled at inference. Additional information such as pitch or energy may also be sampled to provide improved diversity for synthetic speech generation.

What is claimed is: 
     
         1 . A computer-implemented method, comprising:
 updating one or more parameters of one or more machine learning systems using, at least, a first distribution corresponding to a first set of audio samples and a second distribution corresponding to a second set of synthesized audio samples; and   removing, after the updating of the one or more parameters of the one or more machine learning systems, the second distribution to form an inferencing distribution.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 generating the second set of synthesized audio samples; and   generating the second distribution for the second set of synthesized audio samples.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein generating the second set of synthesized audio samples further comprises:
 modifying one or more features of at least a subset of the first set of audio samples.   
     
     
         4 . The computer-implemented method of  claim 1 , wherein the second set of synthesized audio samples includes at least one audio sample from the first set of audio samples. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein one or more identifiers are applied to the second set of synthesized audio samples. 
     
     
         6 . The computer-implemented method of  claim 1 , further comprising:
 inferencing a generated audio clip, wherein the inferencing is performed using the inferencing distribution.   
     
     
         7 . The computer-implemented method of  claim 1 , wherein the first distribution includes a phoneme distribution. 
     
     
         8 . The computer-implemented method of  claim 7 , wherein a phoneme duration for sampling from the phoneme distribution is determined at inference. 
     
     
         9 . A processor, comprising:
 one or more processing units to:
 update one or more parameters of one or more text-to-speech (TTS) machine learning systems using one or more synthetic training clips; 
 identify one or more locations within a distribution corresponding to the one or more synthetic training clips; 
 remove the one or more locations from the distribution to form an inference distribution; and 
 sample, during inferencing and using the TTS machine learning system, from the inference distribution. 
   
     
     
         10 . The processor of  claim 9 , wherein the one or more TTS machine learning systems form at least a portion of an end-to-end parallel speech synthesis system. 
     
     
         11 . The processor of  claim 10 , wherein the one or more processing units are further to generate the one or more synthetic training clips from a plurality of audio segments. 
     
     
         12 . The processor of  claim 10 , wherein the one or more synthetic training clips correspond to augmented audio samples including one or more features augmented based at least on an augmentation probability. 
     
     
         13 . The processor of  claim 12 , wherein the one or more features correspond to at least one of pitch or energy. 
     
     
         14 . The processor of  claim 9 , wherein the one or more processing units are further to use the trained TTS system to generate synthetic speech based at least on an input text sequence. 
     
     
         15 . The processor of  claim 9 , wherein the one or more processing units are further to generate the distribution using at least the one or more synthetic training clips. 
     
     
         16 . A system comprising:
 one or more processing units to perform, based at least on an inferencing distribution, inferencing using one or more machine learning models, wherein the inferencing distribution is generated, at least, from a training distribution of the one or more machine learning models that omits one or more distribution values corresponding to a set of synthesized audio samples after updating one or more parameters of the one or more machine learning models that were determined using the training distribution.   
     
     
         17 . The system of  claim 16 , wherein the training distribution includes distribution values for a first distribution corresponding to a set of audio samples and the set of synthesized audio samples. 
     
     
         18 . The system of  claim 17 , wherein the one or more processing units are further to generate the set of synthesized audio samples. 
     
     
         19 . The system of  claim 16 , wherein the one or more processing units are further to use the one or more machine learning models to generate synthetic speech based at least on an input text sequence. 
     
     
         20 . The system of  claim 16 , wherein the one or more machine learning models form at least a portion of an end-to-end parallel speech synthesis system.