Facial animation control by automatic generation of facial action units using text and speech

Abstract

Text and speech from a computer simulation are processed by a machine learning engine to animate the face of a computer avatar.

What is claimed is: 
     
       1. An apparatus comprising:
 at least one processor configured with instructions to: 
 identify an image of a face of a computerized avatar; 
 identify first modality data related to the avatar, the first modality data comprising speech; 
 identify second modality data related to the avatar, the second modality data comprising text; 
 receive first information based at least in part on the first modality data and second information based at least in part on the second modality data useful for animating the image of the face of the avatar; and 
 animate the face of the avatar in accordance with both the first and second information, wherein the instructions are executable to access at least one machine learning (ML) model to animate the face of the avatar, the ML model being executable to receive anchor points derived from an image of the face of the avatar to generate facial action units (FAUs) and to associate the speech and text with the FAU; 
 the instructions being executable to temporally align the speech and text with each other at least in part using phoneme boundaries extracted from the speech, the instructions being further executable to align in time emotions and degrees of emotions with the speech and text using the phoneme boundaries extracted from the speech. 
 
     
     
       2. The apparatus of  claim 1 , wherein the instructions are executable to derive the emotions from the first and second modality data. 
     
     
       3. The apparatus of  claim 2 , wherein the information is based at least in part on time-aligned word level emotion probabilities produced from the emotions. 
     
     
       4. A method, comprising:
 generating an image of a first face to be animated to speak words in accordance with both first text and first speech; 
 aligning in time the first text and the first speech; 
 animating the image of the first face to speak first words in accordance with the first text and the first speech; wherein 
 the image of the first face is animated at least in part by processing a sliding window of words in the first speech such that words 1 through N in the first speech are associated with at least a first emotion and the image of the first face is animated according to the first emotion, and words 2 through N+M in the first speech are associated with at least a second emotion and the image of the first face is animated according to the second emotion, wherein N is an integer greater than two and M is an integer. 
 
     
     
       5. The method of  claim 4 , comprising:
 training a machine learning (ML) model using a training set of animated faces speaking known words; 
 inputting the first text and first speech to the ML model; 
 animating the image of the first face in accordance with output of the ML model; 
 detecting emotion and sentiment from the first text; 
 aligning the first text with speech representing the first text to render aligned text/speech; and 
 inputting the emotion, sentiment, and aligned text/speech to the ML model. 
 
     
     
       6. The method of  claim 5 , comprising:
 inputting a target emotion to the ML model. 
 
     
     
       7. The method of  claim 5 , comprising:
 receiving first probabilities from the ML model representing facial action. 
 
     
     
       8. The method of  claim 7 , comprising:
 receiving second probabilities from the ML model representing emotion. 
 
     
     
       9. The method of  claim 8 , comprising:
 using one or both of the first and second probabilities to establish facial action units (FAU). 
 
     
     
       10. The method of  claim 9 , comprising:
 animating the image of the first face in accordance with the FAU. 
 
     
     
       11. An assembly comprising:
 at least one display configured to present an animated computer avatar; 
 at least one processor configured with instructions to execute a machine learning (ML) model, the instructions being executable to: 
 receive text indicating speech to be spoken by the avatar; 
 receive speech; 
 align the text and the speech in time at least in part using phoneme boundaries extracted from the speech, the instructions being further executable to align in time emotions with the speech and text using the phoneme boundaries extracted from the speech; 
 process the text and speech using the ML model to generate facial action units (FAU); and 
 animate the computer avatar in accordance with the FAU. 
 
     
     
       12. The assembly of  claim 11 , wherein the instructions are executable to:
 detect emotion and sentiment from the text; 
 align the text with speech representing the text to render aligned text/speech; and 
 inputting the emotion, sentiment, and aligned text/speech to the ML model. 
 
     
     
       13. The assembly of  claim 12 , wherein the instructions are executable to:
 input a target emotion to the ML model. 
 
     
     
       14. The assembly of  claim 11 , wherein the instructions are executable to:
 receive first probabilities from the ML model representing facial action. 
 
     
     
       15. The assembly of  claim 14 , wherein the instructions are executable to:
 receive second probabilities from the ML model representing emotion. 
 
     
     
       16. The assembly of  claim 15 , wherein the instructions are executable to:
 use the first and second probabilities to establish the FAU.