Text-to-speech processing using input voice characteristic data

Abstract

During text-to-speech processing, a speech model creates synthesized speech that corresponds to input data. The speech model may include an encoder for encoding the input data into a context vector and a decoder for decoding the context vector into spectrogram data. The speech model may further include a voice decoder that receives vocal characteristic data representing a desired vocal characteristic of synthesized speech. The voice decoder may process the vocal characteristic data to determine configuration data, such as weights, for use by the speech decoder.

1 - 20 . (canceled) 
     
     
         21 . A computer-implemented method, comprising:
 receiving first input data representing first speech comprising a first plurality of words spoken by a first human voice corresponding to a vocal characteristic;   processing the first input data to determine vocal characteristic data representing at least the vocal characteristic;   receiving second input data corresponding to a speech synthesis task; and   determining synthesized speech data corresponding to synthesized speech comprising the first plurality of words spoken by a synthesized voice corresponding to the vocal characteristic.   
     
     
         22 . The computer-implemented method of  claim 21 , further comprising:
 determining, using an encoder and the second input data, encoded data,   wherein determining the synthesized speech data comprises using the encoded data.   
     
     
         23 . The computer-implemented method of  claim 22 , wherein determining the synthesized speech data further comprises using a machine learning component. 
     
     
         24 . The computer-implemented method of  claim 23 , wherein the machine learning component comprises a decoder, and determining the synthesized speech data comprises processing the encoded data using the decoder. 
     
     
         25 . The computer-implemented method of  claim 21 , further comprising:
 determining, using a trained model and the vocal characteristic data, model output data,   wherein determining the synthesized speech data further uses the model output data.   
     
     
         26 . The computer-implemented method of  claim 25 , further comprising:
 determining, using an encoder and the second input data, encoded data,   wherein determining the synthesized speech data comprises processing the encoded data and the model output data using a machine learning decoder.   
     
     
         27 . The computer-implemented method of  claim 21 , further comprising:
 capturing audio by at least one microphone; and   determining, by the at least one microphone and based at least in part on the audio, the first input data.   
     
     
         28 . The computer-implemented method of  claim 21 , further comprising:
 receiving third input data corresponding to a user input; and   processing the third input data to determine a request to perform the speech synthesis task corresponding to the vocal characteristic.   
     
     
         29 . The computer-implemented method of  claim 28 , wherein processing the third input data comprises:
 performing natural language understanding (NLU) processing using the third input data to determine an intent to perform the speech synthesis task corresponding to the vocal characteristic.   
     
     
         30 . The computer-implemented method of  claim 21 , wherein processing the first input data comprises:
 processing the first input data using a machine learning component to determine the vocal characteristic data.   
     
     
         31 . A system, comprising:
 at least one processor; and   at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:
 receive first input data representing first speech comprising a first plurality of words spoken by a first human voice corresponding to a vocal characteristic; 
 process the first input data to determine vocal characteristic data representing at least the vocal characteristic; 
 receive second input data corresponding to a speech synthesis task; and 
 determine synthesized speech data corresponding to synthesized speech comprising the first plurality of words spoken by a synthesized voice corresponding to the vocal characteristic. 
   
     
     
         32 . The system of  claim 31 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
 determine, using an encoder and the second input data, encoded data,   wherein determination of the synthesized speech data comprises using the encoded data.   
     
     
         33 . The system of  claim 32 , wherein determination of the synthesized speech data further comprises using a machine learning component. 
     
     
         34 . The system of  claim 33 , wherein the machine learning component comprises a decoder, and determination of the synthesized speech data comprises processing the encoded data using the decoder. 
     
     
         35 . The system of  claim 31 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
 determine, using a trained model and the vocal characteristic data, model output data,   wherein determination of the synthesized speech data further uses the model output data.   
     
     
         36 . The system of  claim 35 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
 determine, using an encoder and the second input data, encoded data,   wherein determination of the synthesized speech data comprises processing the encoded data and the model output data using a machine learning decoder.   
     
     
         37 . The system of  claim 31 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
 capture audio by at least one microphone; and   determine, by the at least one microphone and based at least in part on the audio, the first input data.   
     
     
         38 . The system of  claim 31 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
 receive third input data corresponding to a user input; and   process the third input data to determine a request to perform the speech synthesis task corresponding to the vocal characteristic.   
     
     
         39 . The system of  claim 38 , wherein the instructions that cause the system to process the third input data comprises instructions that, when executed by the at least one processor, cause the system to:
 perform natural language understanding (NLU) processing using the third input data to determine an intent to perform the speech synthesis task corresponding to the vocal characteristic.   
     
     
         40 . The system of  claim 31 , wherein the instructions that cause the system to process the first input data comprises instructions that, when executed by the at least one processor, cause the system to:
 process the first input data using a machine learning component to determine the vocal characteristic data.