Information processing method and information processing system for sound synthesis utilizing identification data associated with sound source and performance styles

Abstract

An information processing system includes at least one memory storing a program and at least one processor. The at least one processor implements the program to input a piece of sound source data obtained by encoding a first identification data representative of a sound source, a piece of style data obtained by encoding a second identification data representative of a performance style, and synthesis data representative of sounding conditions into a synthesis model generated by machine learning, and to generate, using the synthesis model, feature data representative of acoustic features of a target sound of the sound source to be generated in the performance style and according to the sounding conditions, and to generate an audio signal corresponding to the target sound using the generated feature data.

What is claimed is: 
     
       1. An information processing method implemented by a computer, the method comprising:
 providing a first piece of sound source data, which has been obtained by encoding first identification data that identifies a first sound source, wherein the first piece of sound source data represents acoustic features of the first sound source, represented as a first embedding vector in a first multidimensional space; 
 providing a first piece of style data, which has been obtained by encoding second identification data that identifies a first performance style, wherein the first piece of style data represents acoustic features of sound generated by the first sound source in the first performance style, represented as a first embedding vector in a second multidimensional space; 
 generating, using a synthesis model generated by machine learning, first feature data representative of acoustic features of a first target sound of the first sound source to be generated in the first performance style and according to first sound conditions, by inputting into the synthesis model:
 the first piece of sound source, 
 the first piece of style data, and 
 first synthesis data representative of the first sounding conditions; and 
 
 generating a first audio signal corresponding to the first target sound using the generated first feature data. 
 
     
     
       2. The information processing method according to  claim 1 , further comprising:
 providing a second piece of sound source data, which has been obtained by encoding third identification data that identifies a second sound source, wherein the second piece of sound source data represents acoustic features of the second sound source, represented as a second embedding vector in the first multidimensional space; 
 providing a second piece of style data, which has been obtained by encoding fourth identification data that identifies a second performance style, wherein the second piece of style data represents acoustic features of sound generated by the second sound source in the second performance style, represented as a second embedding vector in the second multidimensional space; 
 generating, using the synthesis model, second feature data representative of acoustic features of a second target sound of the second sound source to be generated in the second performance style and according to second sounding conditions, by inputting into the synthesis model:
 the second piece of sound source data, 
 the second piece of style data, and 
 second synthesis data representative of the second sounding conditions; 
 
 generating a second audio signal corresponding to the second target sound using the generated second feature data; and 
 updating the second sound source data and the synthesis model to decrease a difference between known feature data and the second feature data, wherein the known feature data relates to sound generated by the generated second audio signal. 
 
     
     
       3. The information processing method according to  claim 2 , wherein:
 the first embedding vector and the second embedding vector in the first multidimensional space represent relations between acoustic features of sounds generated by different sound sources, and 
 the first embedding vector and the second embedding vector in the second multidimension space represent relations between acoustic features of sounds generated in different performance styles. 
 
     
     
       4. The information processing method according to  claim 2 , wherein:
 the first performance style represented by the first piece of style data includes a first sound-output environment and a first recording environment, and 
 the second performance style represented by the second piece of style data includes a second sound-output environment and a second recording environment. 
 
     
     
       5. The information processing method according to  claim 1 , wherein the synthesis model includes:
 a first generative model configured to generate a series of fundamental frequencies of the first target sound; and 
 a second generative model configured to generate a series of spectrum envelopes of the first target sound in accordance with the series of fundamental frequencies generated by the first generative model. 
 
     
     
       6. The information processing method according to  claim 5 , further comprising:
 editing the series of fundamental frequencies generated by the first generative model in response to an instruction from a user, 
 wherein the second generative model generates the series of spectrum envelopes of the first target sound in accordance with the edited series of fundamental frequencies. 
 
     
     
       7. The information processing method according to  claim 1 , wherein the first sounding conditions include a pitch of each note included in the first synthesis data. 
     
     
       8. The information processing method according to  claim 1 , wherein the first sounding conditions include a phonetic identifier of each note included in the first synthesis data. 
     
     
       9. The information processing method according to  claim 1 , wherein the first piece of sound source data to be input into the synthesis model is selected by a user from among a plurality of pieces of sound source data, each piece corresponding to a different sound source. 
     
     
       10. The information processing method according to  claim 1 , wherein the first piece of style data to be input into the synthesis model is selected by a user from among a plurality of pieces of style data, each piece corresponding to a different performance style. 
     
     
       11. The information processing method according to  claim 1 , wherein:
 the first identification data represents a first series of numeric values, and 
 the second identification data represents a second series of numeric values. 
 
     
     
       12. The information processing method according to  claim 1 , wherein:
 the first sound source is a user, and 
 the acoustic features of the first sound source represent voice qualities of the user. 
 
     
     
       13. An information processing system comprising:
 at least one memory storing a program; and 
 at least one processor that implements the program to:
 provide a piece of sound source data, which has been obtained by encoding first identification data that identifies a sound source, wherein the piece of sound source data represents acoustic features of the sound source, represented as an embedding vector in a first multidimensional space; 
 provide a piece of style data, which has been obtained by encoding second identification data that identifies a performance style, wherein the piece of style data represents acoustic features of sound generated by the sound source in the performance style, represented as an embedding vector in a second multidimensional space; 
 generate, using a synthesis model generated by machine learning, feature data representative of acoustic features of a target sound of the sound source to be generated in the performance style and according to sound conditions, by inputting into the synthesis model:
 the piece of sound source data, 
 the piece of style data, and 
 
 generate an audio signal corresponding to the target sound using the generated feature data. 
 
 
     
     
       14. A non-transitory medium storing a program executable by a computer to execute a method comprising:
 providing a piece of sound source data by encoding first identification data that identifies a sound source, wherein the piece of sound source data represents acoustic features of the sound source, represented as an embedding vector in a first multidimensional space; 
 providing a piece of style data by encoding second identification data that identifies a performance style, wherein the piece of style data represents acoustic features of sound generated by the sound source in the performance style, represented as an embedding vector in a second multidimensional space; 
 generating, using a synthesis model generated by machine learning, feature data representative of acoustic features of a target sound of the sound source to be generated in the performance style and according to sound conditions, by inputting into the synthesis model:
 the piece of sound source data, 
 the piece of style data, and 
 synthesis data representative of the sounding conditions; and 
 
 generating an audio signal corresponding to the target sound using the generated feature data.