Text-to speech conversion system for synchronizing between synthesized speech and a moving picture in a multimedia environment and a method of the same

Abstract

The present invention provides a text-to-speech conversion system (TTS) for interlocking synchronizing with multimedia and a method for organizing input data of the TTS which can enhance the natural naturalness of synthesized speech and accomplish the synchronization of multimedia with TTS by defining additional prosody information, the information required to interlock synchronize TTS with multimedia, and interface between these this information and TTS for use in the production of the synthesized speech.

1. A text-to-speech conversion system for interlocking with multimedia comprising;
 a multimedia information input unit for organizing text, prosody information, information on synchronization with a moving picture, lip-shape information, picture information, and individual property information including a gender, age, accent, pronunciation and speech rate of synthesized speech; 
 a data distributor for distributing the information from said multimedia information input unit into information for each media; 
 a language processor for converting the text distributed by said data distributor into a phoneme stream, presuming prosody information and symbolizing the presumed prosody information; 
 a prosody processor for calculating a prosody control parameter value from the symbolized prosody information from the language processor; 
 a synchronization adjuster for adjusting a duration of each phoneme using the synchronization information distributed by said data distributor; 
 a synthesis unit database for receiving the individual property information from said data distributor, selecting synthesis units adaptable to gender and age and outputting data required for synthesis; 
 a signal processor for producing a synthesized speech using the prosody control parameter and the data output from said synthesis unit database; and 
 a picture output apparatus for outputting the picture information distributed by said data distributor onto a screen. 
 
     
     
       2. A method for organizing input data of a text-to-speech conversion system for interlocking with multimedia, said method comprising the steps of:
 (a) classifying multimedia input information organized for enhancing natural synthesized speech and implementing synchronization of multimedia with text-to-speech into text, prosody information, information on synchronization with a moving picture, lip-shaped information, picture information, and individual property information using a multimedia information input unit; 
 (b) distributing using a data distributor the multimedia input information classified in the multimedia information input unit based on respective information; 
 (c) converting the text distributed by the data distributor into a phoneme stream, presuming prosody information and symbolizing the presumed prosody information using a language processor; 
 (d) calculating a prosody control parameter value which is not included in the multimedia input information using a prosody processor; 
 (e) adjusting a duration of each phoneme using a synchronization adjuster so as to synchronize a processing result of the prosody processor with a picture signal according to the synchronization information distributed by the data distributor; 
 (f) selecting synthesis units adaptable to gender and age based on the individual property information from the data distributor using a synthesis unit database and outputting data required for synthesis; 
 (g) producing synthesized speech using a signal processor based on the prosody information distributed by the data distributor, a processing result of the synchronization adjuster, and the data from the synthesis unit database; and 
 (h) outputting the picture information distributed by the data distributor onto a screen using a picture output unit. 
 
     
     
       3. The method in accordance with  claim 2 , wherein the organized multimedia information comprises text information, prosody information, information on synchronization with a moving picture, lip-shaped information, and individual property information. 
     
     
       4. The method in accordance with  claim 3 , wherein the prosody information comprises a number of phoneme, phoneme stream information, duration of each phoneme, pitch pattern of the phoneme, and energy pattern of the phoneme. 
     
     
       5. The method in accordance with  claim 4 , wherein the duration time of the phoneme is indicative of a value of pitch at a beginning point, a mid point, and an end point within the phoneme. 
     
     
       6. The method in accordance with  claim 5 , wherein the energy pattern of the phoneme is indicative of a value of energy in decibels at the beginning point, the mid point, and the end point within the phoneme. 
     
     
       7. The method in accordance with  claim 3 , wherein the synchronization information comprises text, lip-shape, location information with a moving picture, and duration information. 
     
     
       8. The method in accordance with  claim 3 , wherein the synchronization information comprises a beginning point, duration and delay time information of a starting point, and duration of each phoneme is controlled by the synchronization information. 
     
     
       9. The method in accordance with  claim 3 , wherein the synchronization information is composed of a duration of a beginning point of a sentence, a duration information of a starting point, and duration of each phoneme is controlled by forecast lip-shape considered an articulation manner of the phoneme and articulation control of lip-shape within the synchronization and duration information of the synchronization information. 
     
     
       10. The method in accordance with  claim 3 , wherein the synthesized speech is produced based on beginning point information, end point information, and phoneme information for each phoneme within an interval associated with a speech signal. 
     
     
       11. The method in accordance with  claim 3 , wherein the synthesized speech is produced based on a distance of an opening between an upper lip and a lower lip, a distance between end points of the lips, and an extent of projection of a lip, and a lip-shape quantized and normalized pattern is defined depending on articulation location and articulation manner of the phoneme on a basis of pattern with discriminative property. 
     
     
       12. The method in accordance with  claim 3 , wherein if the multimedia input information comprises prosody information, further comprising the steps of:
 (i) converting the prosody information into a data structure recognizable by the signal processor; and 
 (j) transmitting the converted prosody information to the prosody processor and the synchronization adjustor. 
 
     
     
       13. The method in accordance with  claim 3 , wherein if the multimedia input information includes individual property information, further comprising the steps of:
 (k) converting the individual property information into a data structure recognizable by the synthesis unit database and the prosody processor within the text-to-speech; 
 (l) transmitting the converted individual property information to the synthesis unit database and the prosody processor. 
 
     
     
       14. A text-to-speech conversion system (TTS) for synchronizing synthesized speech and a moving picture which is to be displayed on a picture output apparatus which is connected with the TTS, the TTS including a language processor for converting the text into phoneme stream and presuming prosody information from the phoneme stream; a prosody processor for calculating prosody control parameter values from the prosody information using a predefined rule; and a signal processor for producing synthesized speech using the prosody control parameter values and synthetic data stored in a synthesis unit database, characterized in that the TTS comprises:
 a multimedia information input unit for inputting a set of multimedia information, the set of multimedia information including moving picture information, text information, and synchronization information;   a data distributor for classifying the set of multimedia information into a plurality of subsets of the multimedia information to distribute each subset of the multimedia information into a corresponding one of the language processor, prosody processor, signal processor and the picture output apparatus; and   a synchronization adjuster for adjusting the duration of each phoneme in the phoneme stream using the synchronization information subset distributed by said data distributor to synchronize between synthesized speech to be produced by the signal processor and the moving picture to be displayed on the picture output apparatus.   
     
     
       15. A method for synchronizing synthesized speech generated from a TTS and a moving picture which is to be displayed on a picture output apparatus which is connected with the TTS, the method comprising the steps of:
 receiving a set of multimedia information which includes text information, moving picture information and synchronization information;   classifying the set of the received multimedia information into a plurality of subsets of the information including a synchronization information subset;   converting each classified text information subset into a phoneme stream;   presuming prosody information from each phoneme stream;   calculating prosody control parameter values based on the prosody information;   adjusting the duration of each phoneme of each phoneme stream using the respective classified synchronization information subset to synchronize between the synthesized speech and the moving picture; and   producing the synthesized speech using the prosody control parameter values and data in a synthesis unit database, in synchronism with the moving picture to be displayed.   
     
     
       16. The method according the claim 15, wherein said prosody control parameters are comprised of the number of phonemes, duration time of each phoneme, pitch pattern of each phoneme and energy pattern of each phoneme. 
     
     
       17. The method according to claim 16, wherein said pitch pattern of each phoneme is indicative of a value of pitch of the desired synthesized speech at a beginning point, a middle point, and an end point within each phoneme. 
     
     
       18. The method according to claim 16, wherein said energy pattern of each phoneme is indicative of a value of energy of the desired synthesized speech in decibels at a beginning point, a mid point and an end point within each phoneme. 
     
     
       19. The system according to claim 14, wherein said synchronization information subset includes lip-shape information, said lip-shape information including a number of lip-shape change points, a location of lip-shape change points in a sentence and a lip-shape representation at every lip-shape change point. 
     
     
       20. The system according to claim 14, wherein a set of said multimedia information further includes individual property information, said individual property information subset including gender and age information of the synthesized speech. 
     
     
       21. The system according to claim 14, wherein if a set of said multimedia information further includes prosody control parameters, said prosody control parameters are capable of being utilized in said synchronization adjuster without the processing of said language processor and prosody processor. 
     
     
       22. The system according the claim 21, wherein said prosody control parameters include the number of phonemes in the data stream, a duration time of each phoneme, a pitch pattern of each phoneme and an energy pattern of each phoneme. 
     
     
       23. The system according to claim 22, wherein said pitch pattern of each phoneme is indicative of a value of pitch of the desired synthesized speech at a beginning point, a middle point, and an end point within each phoneme. 
     
     
       24. The system according to claim 22, wherein said energy pattern of each phoneme is indicative of a value of energy of the desired synthesized speech in decibels at a beginning point, a mid point and an end point within each phoneme. 
     
     
       25. The method of claim 15, wherein the classified synchronization information subset includes lip-shape information, said lip-shape information including a number of lip-shape change points, a location of lip-shape change points in a sentence and a lip-shape representation at every lip-shape change point. 
     
     
       26. The method according to claim 15, wherein said set of the received multimedia information further includes an individual property information subset, said individual property information subset including gender and age information of the synthesized speech. 
     
     
       27. A method for synchronizing synthesized speech generated from a TTS and a moving picture which is to be displayed on a picture output apparatus which is connected with the TTS, the method comprising the steps of:
 receiving a set of multimedia information which includes text information, moving picture information, synchronization information and prosody control parameters, said prosody control parameters including a duration of each phoneme;   classifying a set of the received multimedia information into a plurality of subsets of the information including a synchronization information subset;   adjusting the duration of each phoneme using the classified synchronization information subset to synchronize between the synthesized speech and the moving picture; and   producing the synthesized speech using the prosody control parameter values included in a set of the received multimedia information and data in a synthesis unit database, in synchronism with the moving picture to be displayed onto screen in the picture output apparatus.   
     
     
       28. A process for producing synthesized speech in synchronism with an associated moving picture characterized in that the process comprises receiving a set of multimedia information including text information, moving picture information and synchronization information; and synthesizing speech from the received text information in synchronization with the received moving picture information using the received synchronization information. 
     
     
       29. A speech synthesizer for use in synchronizing synthesized speech generated from a TTS and a moving picture which is to be displayed on a picture output apparatus which is connected with the TTS, the speech synthesizer comprising:
 means for receiving prosody control parameters including a duration of each phoneme, synchronization information and moving picture data;   means for adjusting the duration of each phoneme of each phoneme stream using the synchronization information to synchronize between the synthesized speech and the moving picture; and   means for producing the synthesized speech using the prosody control parameter values and data in a synthesis unit database, in synchronism with the moving picture to be displayed.   
     
     
       30. A synthesizer for producing a synthesized speech using a text information, comprising:
 receiving means for receiving the text information usable for synthesizing speech, and synchronization information including a number of lip-shape change points and a lip-shape representation at every lip-shape change point;   synthesizing means, for producing the synthesized speech from the text information using the synchronization information; and   outputting means for outputting the synthesized speech.   
     
     
       31. The synthesizer according to claim 30, wherein a picture output apparatus is connected to the synthesizer and operated in synchronization with the synthesizer. 
     
     
       32. The synthesizer according to claim 31, wherein the synchronization information is related to a moving picture outputted from the picture output apparatus.