Synthetic audio output method and apparatus, storage medium, and electronic device

Abstract

This application provide a synthetic audio output method and apparatus, a storage medium, and an electronic device. The method includes: inputting input text and a specified target identity identifier into an audio output model; extracting an identity feature sequence of a target identity by an identity recognition model; extracting a phoneme feature sequence corresponding to the input text by an encoding layer of a speech synthesis model; superimposing and inputting the identity feature sequence of the target identity and the phoneme feature sequence into a variable adapter of the speech synthesis model; and after duration prediction and alignment, energy prediction, and pitch prediction are performed on the phoneme feature sequence by the variable adapter, outputting a target Mel-frequency spectrum feature corresponding to the input text through a decoding layer of the speech synthesis model; and inputting the target Mel-frequency spectrum feature into a vocoder to output synthetic audio.

What is claimed is: 
     
       1. A synthetic audio output method, comprising:
 inputting input text and a specified target identity identifier into an audio output model, wherein the target identity identifier uniquely corresponds to a target identity, the audio output model is a neural network model trained by using sample data, and the audio output model comprises an identity recognition model and a speech synthesis model; 
 extracting an identity feature sequence of the target identity by the identity recognition model, wherein the identity recognition model saves an identity mapping table during training, and the identity mapping table contains mapping between an identity identifier and the identity feature sequence; 
 extracting a phoneme feature sequence corresponding to the input text by an encoding layer of the speech synthesis model; 
 superimposing and inputting, the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text, into a variable adapter of the speech synthesis model; and after duration prediction and alignment, energy prediction, and pitch prediction are performed on the phoneme feature sequence by the variable adapter, performing decoding by a decoding layer of the speech synthesis model and outputting a target Mel-frequency spectrum feature corresponding to the input text, wherein the target Mel-frequency spectrum feature conforms to a speaking style of the target identity; and 
 inputting the target Mel-frequency spectrum feature into a vocoder to output synthetic audio conforming to the speaking style of the target identity through the vocoder, 
 wherein before the inputting input text and a specified target identity identifier into an audio output model, the method comprises: 
 training a preliminary model of the identity recognition model by using a first training sample, to obtain a first identity recognition model; 
 training a preliminary model of the speech synthesis model by using a second training sample, to obtain a first speech synthesis model; and 
 training both the first identity recognition model and the first speech synthesis model by using a third training sample, to obtain the audio output model, 
 wherein the training a preliminary model of the speech synthesis model by using a second training sample, to obtain a first speech synthesis model comprises: 
 inputting the second training sample into the preliminary model of the speech synthesis model, wherein the second training sample comprises second sample text and second sample audio recorded by a speaker by the second sample text as content, and the preliminary model of the speech synthesis model comprises a feature encoding module, a variable adaptation module, and a feature decoding module, wherein the variable adaptation module comprises a duration prediction layer, a length adjustment layer, an energy prediction layer, and a pitch prediction layer; 
 acquiring a phoneme feature sequence corresponding to the second sample text and an audio feature sequence corresponding to second sample audio data, wherein the phoneme feature sequence corresponding to the second sample text is extracted by the feature encoding module, and the audio feature sequence corresponding to the second sample audio is pre-processed through a data preprocessing step; 
 performing duration prediction on the phoneme feature sequence corresponding to the second sample text by the duration prediction layer, and comparing a result of the duration prediction with the audio feature sequence corresponding to the second sample audio data, to obtain a first loss function; 
 inputting, into the duration prediction layer, the phoneme feature sequence corresponding to the second sample text and the audio feature sequence corresponding to the second sample audio data as first data, wherein, through the duration prediction layer, a low dimensional average value and a variance for the input first data are calculated to obtain second data, the second data is sampled from a latent variable space to obtain potential data, dimensionality augmentation is performed on the potential data to obtain third data, the first data is compared with the third data to obtain a second loss function, and probability distribution of the third data is compared with standard normal distribution to obtain a third loss function, wherein the first loss function, the second loss function, and the third loss function jointly adjust a parameter of the duration prediction layer based on a specific weight; 
 inputting data output from the duration prediction layer into the length adjustment layer, and performing alignment based on Gaussian distribution probability during a data stretching process; 
 inputting the audio feature sequence corresponding to the second sample audio data into the energy prediction layer and the pitch prediction layer respectively, to output an energy feature sequence and a pitch feature sequence; 
 superimposing data output from the length adjustment layer, the energy prediction layer, and the pitch prediction layer, and then inputting the data into the feature decoding module to output a Mel-frequency spectrum feature through the feature decoding module; and 
 verifying a difference between the Mel-frequency spectrum feature output from the feature decoding module and the audio feature sequence corresponding to the second sample audio data through a loss function, and updating a model parameter of the preliminary model of the speech synthesis model to obtain the first speech synthesis model when the difference between the Mel-frequency spectrum feature output from the feature decoding module and the audio feature sequence corresponding to the second sample audio data is less than a second preset threshold through a plurality of iterations. 
 
     
     
       2. The synthetic audio output method according to  claim 1 , wherein the identity recognition model is a model constructed based on a residual network; during the training, an input audio feature is extracted by a convolutional residual module as a main body, frame-level input is converted to a speech-level speaker feature by a feature averaging module, through linear transformation, temporarily collected speaker features are mapped by a standardization module to an identity feature sequence corresponding to an identity identifier of a speaker, and the identity recognition model is trained by a triple loss function to maximize cosine similarity between same speakers and minimize cosine similarity between different speakers. 
     
     
       3. The synthetic audio output method according to  claim 1 , wherein the training a preliminary model of the identity recognition model by using a first training sample, to obtain a first identity recognition model comprises:
 inputting the first training sample into the preliminary model of the identity recognition model, wherein the first training sample comprises first sample audio containing audio of multiple speakers and an identity tag corresponding to each speaker, and the preliminary model of the identity recognition model comprises at least two convolutional residual modules, an averaging module, an affine module, a standardization module, and a loss function module, wherein the convolutional residual module includes a first convolution layer and a first residual layer including at least four convolution sublayers and one activation function sublayer; 
 outputting a frame-level audio feature corresponding to first sample audio data by the convolutional residual module, performing zero centering and variance normalization on the frame-level audio feature, and then inputting the frame-level audio feature into the averaging module; 
 averaging received data by the averaging module so that audio with a specific length corresponds to an audio feature with a specific length, and then inputting the averaged data into the affine module; 
 performing dimensionality reduction on the received data by the affine module, and then mapping the data on which the dimensionality reduction is performed to an identity feature sequence representing an identity of the speaker by the standardization module; and 
 verifying a difference between the identity feature sequence and the identity tag by the loss function module, and updating a model parameter of the preliminary model of the identity recognition model to obtain the first identity recognition model when the difference between the identity feature sequence and the identity tag is less than a first preset threshold through a plurality of iterations. 
 
     
     
       4. The synthetic audio output method according to  claim 1 , wherein the training both the first identity recognition model and the first speech synthesis model by using a third training sample, to obtain the audio output model comprises:
 inputting the third training sample into the preliminary model of the audio output model, wherein the preliminary model of the audio output model comprises the first identity recognition model and the first speech synthesis model, the third training sample comprises third sample audio and third sample text corresponding to the third sample audio, and the third sample audio contains audio of a target identity group; 
 extracting a phoneme feature sequence from the third sample text by the first speech synthesis model, and converting the phoneme feature sequence into a phonemic latent variable feature by a latent variable space; 
 extracting an identity feature from each piece of the third sample audio by the first identity recognition model, and encoding an identity identifier corresponding to each identity feature to obtain an identity identifier feature; 
 superimposing the phonemic latent variable feature, the identity feature and the identity identifier feature, and a fundamental frequency feature, a duration feature and an energy feature corresponding to the third sample audio to obtain a latent variable sequence, and training a variable adapter module of the first speech synthesis model by using the latent variable sequence; and 
 verifying a difference between a Mel-frequency spectrum feature output from the first speech synthesis model and an audio feature corresponding to third sample audio data through a loss function, and updating a model parameter of the preliminary model of the audio output model to obtain the audio output model when the difference between the Mel-frequency spectrum feature output from the first speech synthesis model and the audio feature corresponding to the third sample audio data is less than a third preset threshold through a plurality of iterations. 
 
     
     
       5. A non-transitory computer readable storage medium, in which a computer program is stored, and the computer program is configured to implement a synthetic audio output method while being run,
 wherein the synthetic audio output method comprises: 
 inputting input text and a specified target identity identifier into an audio output model, wherein the target identity identifier uniquely corresponds to a target identity, the audio output model is a neural network model trained by using sample data, and the audio output model comprises an identity recognition model and a speech synthesis model; 
 extracting an identity feature sequence of the target identity by the identity recognition model, wherein the identity recognition model saves an identity mapping table during training, and the identity mapping table contains mapping between an identity identifier and the identity feature sequence; 
 extracting a phoneme feature sequence corresponding to the input text by an encoding layer of the speech synthesis model; 
 superimposing and inputting, the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text, into a variable adapter of the speech synthesis model; and after duration prediction and alignment, energy prediction, and pitch prediction are performed on the phoneme feature sequence by the variable adapter, performing decoding by a decoding layer of the speech synthesis model and outputting a target Mel-frequency spectrum feature corresponding to the input text, wherein the target Mel-frequency spectrum feature conforms to a speaking style of the target identity; and 
 inputting the target Mel-frequency spectrum feature into a vocoder to output synthetic audio conforming to the speaking style of the target identity through the vocoder, 
 wherein before the inputting input text and a specified target identity identifier into an audio output model, the method comprises: 
 training a preliminary model of the identity recognition model by using a first training sample, to obtain a first identity recognition model; 
 training a preliminary model of the speech synthesis model by using a second training sample, to obtain a first speech synthesis model; and 
 training both the first identity recognition model and the first speech synthesis model by using a third training sample, to obtain the audio output model, 
 wherein the training a preliminary model of the speech synthesis model by using a second training sample, to obtain a first speech synthesis model comprises: 
 inputting the second training sample into the preliminary model of the speech synthesis model, wherein the second training sample comprises second sample text and second sample audio recorded by a speaker by the second sample text as content, and the preliminary model of the speech synthesis model comprises a feature encoding module, a variable adaptation module, and a feature decoding module, wherein the variable adaptation module comprises a duration prediction layer, a length adjustment layer, an energy prediction layer, and a pitch prediction layer; 
 acquiring a phoneme feature sequence corresponding to the second sample text and an audio feature sequence corresponding to second sample audio data, wherein the phoneme feature sequence corresponding to the second sample text is extracted by the feature encoding module, and the audio feature sequence corresponding to the second sample audio is pre-processed through a data preprocessing step; 
 performing duration prediction on the phoneme feature sequence corresponding to the second sample text by the duration prediction layer, and comparing a result of the duration prediction with the audio feature sequence corresponding to the second sample audio data, to obtain a first loss function; 
 inputting, into the duration prediction layer, the phoneme feature sequence corresponding to the second sample text and the audio feature sequence corresponding to the second sample audio data as first data, wherein, through the duration prediction layer, a low dimensional average value and a variance for the input first data are calculated to obtain second data, the second data is sampled from a latent variable space to obtain potential data, dimensionality augmentation is performed on the potential data to obtain third data, the first data is compared with the third data to obtain a second loss function, and probability distribution of the third data is compared with standard normal distribution to obtain a third loss function, wherein the first loss function, the second loss function, and the third loss function jointly adjust a parameter of the duration prediction layer based on a specific weight; 
 inputting data output from the duration prediction layer into the length adjustment layer, and performing alignment based on Gaussian distribution probability during a data stretching process; 
 inputting the audio feature sequence corresponding to the second sample audio data into the energy prediction layer and the pitch prediction layer respectively, to output an energy feature sequence and a pitch feature sequence; 
 superimposing data output from the length adjustment layer, the energy prediction layer, and the pitch prediction layer, and then inputting the data into the feature decoding module to output a Mel-frequency spectrum feature through the feature decoding module; and 
 verifying a difference between the Mel-frequency spectrum feature output from the feature decoding module and the audio feature sequence corresponding to the second sample audio data through a loss function, and updating a model parameter of the preliminary model of the speech synthesis model to obtain the first speech synthesis model when the difference between the Mel-frequency spectrum feature output from the feature decoding module and the audio feature sequence corresponding to the second sample audio data is less than a second preset threshold through a plurality of iterations. 
 
     
     
       6. An electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to implement a synthetic audio output method,
 wherein the synthetic audio output method comprises: 
 inputting input text and a specified target identity identifier into an audio output model, wherein the target identity identifier uniquely corresponds to a target identity, the audio output model is a neural network model trained by using sample data, and the audio output model comprises an identity recognition model and a speech synthesis model; 
 extracting an identity feature sequence of the target identity by the identity recognition model, wherein the identity recognition model saves an identity mapping table during training, and the identity mapping table contains mapping between an identity identifier and the identity feature sequence; 
 extracting a phoneme feature sequence corresponding to the input text by an encoding layer of the speech synthesis model; 
 superimposing and inputting, the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text, into a variable adapter of the speech synthesis model; and after duration prediction and alignment, energy prediction, and pitch prediction are performed on the phoneme feature sequence by the variable adapter, performing decoding by a decoding layer of the speech synthesis model and outputting a target Mel-frequency spectrum feature corresponding to the input text, wherein the target Mel-frequency spectrum feature conforms to a speaking style of the target identity; and 
 inputting the target Mel-frequency spectrum feature into a vocoder to output synthetic audio conforming to the speaking style of the target identity through the vocoder, 
 wherein before the inputting input text and a specified target identity identifier into an audio output model, the method comprises: 
 training a preliminary model of the identity recognition model by using a first training sample, to obtain a first identity recognition model; 
 training a preliminary model of the speech synthesis model by using a second training sample, to obtain a first speech synthesis model; and 
 training both the first identity recognition model and the first speech synthesis model by using a third training sample, to obtain the audio output model, 
 wherein the training a preliminary model of the speech synthesis model by using a second training sample, to obtain a first speech synthesis model comprises: 
 inputting the second training sample into the preliminary model of the speech synthesis model, wherein the second training sample comprises second sample text and second sample audio recorded by a speaker by the second sample text as content, and the preliminary model of the speech synthesis model comprises a feature encoding module, a variable adaptation module, and a feature decoding module, wherein the variable adaptation module comprises a duration prediction layer, a length adjustment layer, an energy prediction layer, and a pitch prediction layer; 
 acquiring a phoneme feature sequence corresponding to the second sample text and an audio feature sequence corresponding to second sample audio data, wherein the phoneme feature sequence corresponding to the second sample text is extracted by the feature encoding module, and the audio feature sequence corresponding to the second sample audio is pre-processed through a data preprocessing step; 
 performing duration prediction on the phoneme feature sequence corresponding to the second sample text by the duration prediction layer, and comparing a result of the duration prediction with the audio feature sequence corresponding to the second sample audio data, to obtain a first loss function; 
 inputting, into the duration prediction layer, the phoneme feature sequence corresponding to the second sample text and the audio feature sequence corresponding to the second sample audio data as first data, wherein, through the duration prediction layer, a low dimensional average value and a variance for the input first data are calculated to obtain second data, the second data is sampled from a latent variable space to obtain potential data, dimensionality augmentation is performed on the potential data to obtain third data, the first data is compared with the third data to obtain a second loss function, and probability distribution of the third data is compared with standard normal distribution to obtain a third loss function, wherein the first loss function, the second loss function, and the third loss function jointly adjust a parameter of the duration prediction layer based on a specific weight; 
 inputting data output from the duration prediction layer into the length adjustment layer, and performing alignment based on Gaussian distribution probability during a data stretching process; 
 inputting the audio feature sequence corresponding to the second sample audio data into the energy prediction layer and the pitch prediction layer respectively, to output an energy feature sequence and a pitch feature sequence; 
 superimposing data output from the length adjustment layer, the energy prediction layer, and the pitch prediction layer, and then inputting the data into the feature decoding module to output a Mel-frequency spectrum feature through the feature decoding module; and 
 verifying a difference between the Mel-frequency spectrum feature output from the feature decoding module and the audio feature sequence corresponding to the second sample audio data through a loss function, and updating a model parameter of the preliminary model of the speech synthesis model to obtain the first speech synthesis model when the difference between the Mel-frequency spectrum feature output from the feature decoding module and the audio feature sequence corresponding to the second sample audio data is less than a second preset threshold through a plurality of iterations.