Unsupervised alignment for text to speech synthesis using neural networks

Abstract

Generation of synthetic speech from an input text sequence may be difficult when durations of individual phonemes forming the input text sequence are unknown. A predominantly parallel process may model speech rhythm as a separate generative distribution such that phoneme duration may be sampled at inference. Additional information such as pitch or energy may also be sampled to provide improved diversity for synthetic speech generation.

What is claimed is: 
     
         1 . A computer-implemented method, comprising:
 determining a phoneme distribution from respective phoneme durations of a plurality of first audio segments;   determining, from at least respective phoneme pitches or respective phoneme energies of the plurality of first audio segments, at least a pitch distribution or an energy distribution;   determining a duration alignment between a sequence of text and a total speech duration using the respective phoneme durations; and   determining a speech alignment for a second audio segment based at least on the duration alignment, the phoneme distribution, and at least one of the pitch distribution or the energy distribution.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 generating a synthesized audio recitation of the sequence of text based on the speech alignment.   
     
     
         3 . The computer-implemented method of  claim 1 , further comprising:
 applying a prior distribution to the duration alignment configured to exclude pairs of phonemes and durations from the plurality of first audio segments that are outside of a specified range.   
     
     
         4 . The computer-implemented method of  claim 3 , wherein the prior distribution is cigar-shaped. 
     
     
         5 . The computer-implemented method of  claim 3 , wherein the specified range is smaller at a start of the duration alignment and at an end of the duration alignment than in a center of the duration alignment. 
     
     
         6 . The computer-implemented method of  claim 3 , wherein the prior distribution is constructed from a beta-binomial distribution. 
     
     
         7 . The computer-implemented method of  claim 1 , wherein the synthesized recitation is generative such that a first synthesized recitation is different from a second synthesized recitation, each of the first synthesized recitation and the second synthesized recitation based on the sequence of text. 
     
     
         8 . The computer-implemented method of  claim 1 , further comprising:
 aligning a plurality of text tokens, from the sequence of text, to respective mel frames, based on the duration alignment.   
     
     
         9 . The computer-implemented method of  claim 8 , further comprising:
 normalizing probability distributions for the plurality of text tokens and the respective mel frames.   
     
     
         10 . A system, comprising:
 one or more processing units to generate, for a sequence of text, a synthetic audio segment comprising a synthesized recitation of the sequence of text based, at least, on a synthetic alignment and at least one distribution, wherein the synthetic alignment is based on phoneme distributions for a plurality of audio segments and a first alignment between the sequence of text and a total speech duration, and further wherein the at least one distribution corresponds to respective phoneme pitches from the plurality of audio segments or respective phoneme energies from the plurality of audio segments.   
     
     
         11 . The system of  claim 10 , wherein the synthetic audio segment is generated by an encoder and a decoder operating in parallel. 
     
     
         12 . The system of  claim 10 , wherein the one or more processing units are further to fit, to the synthetic alignment, a beta-binomial distribution. 
     
     
         13 . The system of  claim 12 , wherein the beta-binomial distribution applies a specified range for probabilities associated with the synthetic alignment. 
     
     
         14 . The system of  claim 10 , wherein the synthesized recitation is generative such that a first synthesized recitation is different from a second synthesized recitation, each of the first synthesized recitation and the second synthesized recitation based on the sequence of text. 
     
     
         15 . The system of  claim 10 , wherein the synthetic alignment is based on an L2 distance between a mel frame at a first time and a text phoneme in the sequence of text. 
     
     
         16 . The system of  claim 10 , wherein the synthetic alignment is based on the phoneme distributions, the respective phoneme pitches, and the respective phoneme energies. 
     
     
         17 . A processor, comprising:
 one or more processing units to:
 determine, from a plurality of audio samples including human speech, alignments between text of the plurality of audio samples, a duration of the plurality of audio samples, and at least one of a pitch of the audio samples or an energy of the audio samples; 
 generate an alignment distribution based on the alignments; 
 determine one or more vectors corresponding to one or more speaker characteristics based on a concentrated probability distribution of a text sequence from the text and mel-frames of the alignment distribution across the duration of the plurality of audio samples. 
   
     
     
         18 . The processor of  claim 17 , wherein the one or more processing units are further to:
 receive an input text sequence; and   generate a synthetic audio clip from the input text sequence based on the one or more vectors.   
     
     
         19 . The processor of  claim 18 , wherein the one or more processing units are further to:
 generate a second synthetic audio clip from the input text sequence, the second synthetic audio clip being different from the synthetic audio clip.   
     
     
         20 . The processor of  claim 17 , wherein the alignment distribution is based, at least in part, on an alignment matrix with a beta-binomial distribution.