Apparatus and method for efficient animation of believable speaking 3d characters in real time

Abstract

An apparatus for animating a moving and speaking enhanced-believability, character in real time, comprising a plurality of behavior generators, each for defining a respective aspect of facial behavior, a unifying scripter, associated with the behavior generators, the scripter operable to combine the behaviors into a unified animation script, and a renderer, associated with the unifying scripter, the renderer operable to render the character in accordance with the script, thereby to enhance believability of the character.

1 . An apparatus for animating a moving and speaking enhanced-believability character in real time, comprising:
 a. a plurality of behavior generators each for defining a respective aspect of facial behavior;   b. a unifying scripter, associated with said behavior generators, said scripter operable to combine said behaviors into a unified animation script; and   c. a renderer, associated with said unifying scripter, said renderer operable to render said character in accordance with said script, thereby to enhance believability of said character.   
     
     
         2 . An apparatus according to  claim 1  wherein said behavior generator is operable to generate said behavior over a defined time. 
     
     
         3 . An apparatus according to  claim 1  wherein said behavior is maintainable in an active behavior pool in accordance with said defined time. 
     
     
         4 . An apparatus according to  claim 2  wherein said behavior comprises a set of behavior elements. 
     
     
         5 . An apparatus according to  claim 4  wherein said behavior elements comprise at least one chosen from a list comprising: geometric; orientation; and appearance elements. 
     
     
         6 . An apparatus according to  claim 4  wherein said behavior elements are operable upon a plurality of polygonal surfaces comprising a visual appearance of said character. 
     
     
         7 . An apparatus according to  claim 3  wherein one of said behavior generators is a facial speech behavior generator operable to generate facial expressions for said character based upon character speech. 
     
     
         8 . An apparatus according to  claim 7  wherein said character speech is at least one chosen from a list comprising a text source and an audio signal. 
     
     
         9 . An apparatus according to  claim 8  wherein said character speech is transformable by a phoneme recognizer, said phoneme recognizer producing pairs of respective phonemes and phoneme weights. 
     
     
         10 . An apparatus according to  claim 9  wherein said phoneme weights effect a relative emphasis of respective phonemes based upon at least one phoneme attribute. 
     
     
         11 . An apparatus according to  claim 10  wherein said phoneme attribute is chosen from a list comprising: amplitude, pitch, and emphasis. 
     
     
         12 . An apparatus according to  claim 8  wherein text from said text source is convertible into a speech audio signal using a text to speech converter. 
     
     
         13 . An apparatus according to  claim 12  wherein said audio signal is forwarded to an audio device for synchronized playback with facial speech behavior of said character. 
     
     
         14 . An apparatus according to  claim 3  wherein one of said behavior generators is an eye gaze direction behavior generator operable to determine eye gaze direction on an animation frame-by-frame basis. 
     
     
         15 . An apparatus according to  claim 14  wherein inputs for said eye gaze direction behavior generator comprise: a primary gaze default direction, probabilities for eye movement, and a maximum time for resetting gaze to said primary default direction. 
     
     
         16 . An apparatus according to  claim 15  wherein said eye gaze direction behavior generator further comprises a primary thresholder operable upon elapsed time since last gaze change versus said maximum time, indicating a gaze change to said primary gaze default direction if elapsed time exceeds said maximum time or indicating no gaze change if said maximum time is not exceeded. 
     
     
         17 . An apparatus according to  claim 16  wherein said eye gaze direction behavior generator further comprises a secondary random thresholder operable in association with said primary thresholder, whereby a random threshold is generated based on said probabilities for eye movement and whereby elapsed time since said last gaze change against said random threshold is compared, said secondary random thresholder thresholding a gaze change by a specific amount in a random direction if elapsed time exceeds said random threshold or no gaze change if said random threshold is not exceeded. 
     
     
         18 . An apparatus according to  claim 17  further comprising a saccade accumulator to simulate a gaze saccade operable in association with said amount of gaze change, said accumulator accumulating said amount of gaze change with an accumulated gaze change amount. 
     
     
         19 . An apparatus according to  claim 18  further comprising a saccade thresholder operable upon said accumulated gaze change amount to determine an actual eye movement equal to said accumulated gaze change amount and to subsequently reset said accumulated gaze change amount to zero when a saccade threshold value is exceeded, and to determine an actual eye movement equal to zero when said saccade threshold value is not exceeded. 
     
     
         20 . An apparatus according to  claim 19  wherein said saccade threshold value is operable upon a maximum up-down and a maximum left-right actual eye movement. 
     
     
         21 . An apparatus according to  claim 20  wherein said maximum up-down and said maximum left-right actual eye movement values are 3 degrees and 7 degrees, respectively. 
     
     
         22 . An apparatus according to  claim 15  wherein a mood and a personality of said character are useable to set said probabilities. 
     
     
         23 . An apparatus according to  claim 3  wherein one of said behavior generators is an eye blink behavior generator, operable to effect eye blink behavior. 
     
     
         24 . An apparatus according to  claim 23  wherein inputs for eye blink behavior generator comprise: an inherent blink rate probability, determinable in association with a personality of said character, speech behavior, from which a speech-related blink probability is generatable; mood, from which a mood-related blink probability is generatable; and eye movement, from which an eye movement related blink probability is generatable. 
     
     
         25 . An apparatus according to  claim 24  wherein said eye blink behavior generator further comprises a summer, operable to sum respectively generated blink probabilities to yield a summed blink probability larger than zero and smaller than one. 
     
     
         26 . An apparatus according to  claim 25  wherein said eye blink behavior generator further comprises a thresholder, operable in association with said summer, to indicate an occurrence of a blink. 
     
     
         27 . An apparatus according to  claim 26  wherein said thresholder comprises:
 a. a random number generator wherein a random number larger than zero and smaller than one is generated;   b. a random thresholder for comparing said summed blink probability against said random number; a blink being indicated if said summed blink probability is larger than said random number, and no blink being indicated if said summed blink probability is smaller than said random number; and   c. an elapsed time thresholder operable in association with said random thresholder indicating a blink status, so that elapsed time since a last blink is compared against a blink threshold time, said elapsed time thresholder indicating a blink if said elapsed time is larger than said blink threshold, or no blink indicated if elapsed time is smaller than said blink threshold.   
     
     
         28 . An apparatus according to  claim 3  wherein one of said behavior generators is a mood behavior generator comprised of
 a. inputs chosen from a list comprising: direct user input, text analysis, and audio analysis;   b. a mood selector for using said inputs in the selection of a mood; and   c. a behavior element producer for generation of at least one behavior element for defining behavior in accordance with said selected mood.   
     
     
         29 . An apparatus according to  claim 28  wherein said direct user input is effectible by means of a graphical input device wherein a plurality of mood selectors are arrayed on a circumference of a circular dial with an adjustable radial hand, said adjustable radial hand being user operable to indicate at least one mood selector and a respective relative mood intensity. 
     
     
         30 . Apparatus according to  claim 29  wherein said adjustable radial hand is user pointable to a desired mood selector. 
     
     
         31 . Apparatus according to  claim 29  wherein said adjustable radial hand is user pointable between two adjacent mood selectors, thereby indicating a relative combination of both moods, based on the relative positioning of said adjustable radial hand between said two adjacent moods. 
     
     
         32 . Apparatus according to  claim 29  wherein the length of said adjustable radial hand is indicative of said relative mood intensity. 
     
     
         33 . An apparatus according to  claim 3  wherein one of said behavior generators is a head orientation behavior generator operable to effect head orientation behavior on an animation frame-by-frame basis. 
     
     
         34 . An apparatus according to  claim 33  wherein said head orientation generator is operable to select from user input and other behavior generator output to use in said effecting of head orientation behavior. 
     
     
         35 . An apparatus according to  claim 33  wherein said head orientation is chosen from a list comprising head turn, nod, and tilt. 
     
     
         36 . An apparatus according to  claim 2  wherein said unifying scripter is operable to order said behaviors according to respective behavior elements and respective defined times, thereby creating said unified animation script. 
     
     
         37 . An apparatus according to  claim 2  wherein a behavior weighter is operable to weigh respective behavior elements using respective weights, to vary respective behavior element influence upon said character. 
     
     
         38 . An apparatus according to  claim 37  wherein said respective weight is
 defined as: Weight=V 0 +f((T−T 0 )/D)*(V 1 −V 0 )) where:   T=a current time;   T 0 =start time, relative to other behavior elements in said set of behavior elements;   D=time duration of said behavior element;   V 1 =weight value when said function evaluates to 1;   V o =weight value when the function evaluates to 0; and   f is a function returning a value ranging from 1 to 0.   
     
     
         39 . Apparatus according to  claim 38  wherein said function is chosen from a list of functions comprising: trigonometriC, wedge, and linear. 
     
     
         40 . An apparatus according to  claim 6  wherein said geometric behavior element comprises changes of position of a plurality of vertices of said polygonal surfaces. 
     
     
         41 . An apparatus according to  claim 6  wherein said orientation behavior element comprises changes to said character resulting from transformations to vertices of said polygonal surfaces. 
     
     
         42 . An apparatus according to  claim 41  wherein said transformations comprise a rigid transformation applied to a plurality of vertices of said polygonal surfaces of at least one part of said character. 
     
     
         43 . An apparatus according to  claim 41  wherein said transformations comprise tapered transformations operable upon a plurality of points of said character in accordance with respective stickiness factors. 
     
     
         44 . An apparatus according to  claim 43  wherein respective stickiness factors are defined as: s v =1−k v  where:
 s v  is said stickiness factor for a rotation of a given vertex within a body of vertices; and   k v  is an angular coefficient for said vertex, k v  further being defined in: θ v,a =k v a where:   a is a rotation of said body of vertices; and   θ v,a  is a resultant rotation at said vertex.   
     
     
         45 . An apparatus according to  claim 44  further comprising a graphics library rotation mechanism to effect a sticky rotation, resembling a standard rigid rotation, said mechanism comprising:
 a. a sorter for sorting said vertices according to respective s v  values;   b. a rotation definer for defining a local rotation, based upon said rotation angle a around an axis W, using library commands;   c. a rotation matrix computer, in association with said sorter and said rotation definer, operable to compute a rigid rotation matrix with angle −s v a for said vertices when s v  is not equal to zero.   d. a vertex multiplier, in association with said thresholder, for multiplying respective vertices with a non-zero s v  value by said rigid rotation matrix, thereby yielding a negative rotation over said angle a.   
     
     
         46 . An apparatus according to  claim 5  wherein said appearance behavior element comprises at least one appearance aspect chosen from a list comprised of: color; texture; and lighting. 
     
     
         47 . An apparatus according to  claim 46  wherein a material is associated with a respective appearance aspect. 
     
     
         48 . A method for animating a moving and speaking enhanced-believability character in real time, comprising the steps of:
 a. defining respective aspects of facial behavior in order to generate respective facial behaviors;   b. unifying said behaviors into a unified animation script; and   c. rendering said character in accordance with said script, thereby to enhance believability of said character.   
     
     
         49 . A method according to  claim 48  wherein said generation of respective behaviors is performed substantially off line. 
     
     
         50 . A method according to  claim 49  wherein said generation of respective behaviors is for a respective defined time. 
     
     
         51 . A method according to  claim 49  wherein said generation of respective behaviors includes generation of facial speech behavior, whereby facial expressions for said character are generated, based upon character speech, comprising the steps of:
 a. using a text source or audio signal as said character speech; and   b. transforming said character speech into phonemes.   
     
     
         52 . A method according to  claim 51  wherein text from said text source is converted into a speech audio signal. 
     
     
         53 . A method according to  claim 52  wherein said audio signal is forwarded for synchronized playback with said facial speech behavior. 
     
     
         54 . A method according to  claim 49  wherein said generation of respective behaviors includes generation of eye gaze direction behavior to determine eye gaze direction on an animation frame-by-frame basis, comprises the steps of:
 a. thresholding elapsed time since last gaze change versus an inputted maximum time;   b. indicating a gaze change to an inputted primary gaze default direction if elapsed time exceeds said maximum time; or, if no gaze change is indicated:   c. thresholding elapsed time since last gaze change versus a randomly generated threshold value, based upon probabilities for eye movement;   d. indicating no gaze change if said maximum time is not exceeded; and   e. indicating a gaze change by a small amount in a random direction if said maximum time is exceeded.   
     
     
         55 . A method according to  claim 54  further comprising the steps of:
 a. summing said gaze change amount to an accumulated gaze change value;   b. thresholding said accumulated gaze change thereby simulating a gaze saccade effect, so that an actual gaze change is set equal to said accumulated gaze change and said accumulated gaze change is subsequently set to zero if said accumulated gaze change exceeds a saccade threshold value; and   c. setting an actual gaze change to zero if said accumulated gaze change does not exceed said saccade threshold value.   
     
     
         56 . A method according to  claim 55  wherein said saccade threshold is definable for an up-down and a left-right direction gaze change. 
     
     
         57 . A method according  claim 54  wherein a mood and a personality of said character are used to set said probabilities. 
     
     
         58 . A method according to  claim 49  wherein said generation of respective behaviors includes generation of eye blink behavior comprising the steps of:
 a. generating a plurality of blink probabilities;   b. summing said blink probabilities to yield a value between zero and one;   c. thresholding said summed blink probability against a randomly generated number between zero and one;   d. indicating a blink if said summed blink probability is larger than said number; or, if no blink is indicated;   e. thresholding an elapsed time since last blink against a blink threshold time;   f. indicating a blink if said elapsed time is larger than said blink threshold time.   
     
     
         59 . A method according to  claim 49  wherein said generation of respective behaviors includes generation of mood behavior comprising using inputs to select said a mood and said mood intensity. 
     
     
         60 . A method according to  claim 49  wherein said generation of respective behaviors includes generation of head orientation behavior on an animation frame-by-frame basis, comprising selecting from user input and other behavior generation to effect head orientation behavior.