Method, device, and computer program product for processing video

Abstract

Methods, devices and computer program products for processing video are disclosed herein. A method includes: generating, based on a reference image and a first frame of a video comprising an object, a two-dimensional avatar image of the object; and generating a base three-dimensional avatar of the object by performing a three-dimensional transformation on the two-dimensional avatar image and the object in the first frame. The method further includes: generating a three-dimensional avatar video corresponding to the video based on the base three-dimensional avatar and features of the video, the features comprising differences of the object between adjacent frames of the video. This solution enables the generation of a customized three-dimensional avatar video for an object in a video, where the avatar can move in synchronization with the object and retain the unique features of the object, and can provide a more detailed and vivid representation than a two-dimensional avatar.

What is claimed is: 
     
         1 . A method for processing video, comprising:
 generating, based on a reference image and a first frame of a video comprising an object, a two-dimensional avatar image of the object;   generating a base three-dimensional avatar of the object by performing a three-dimensional transformation on the two-dimensional avatar image and the object in the first frame; and   generating a three-dimensional avatar video corresponding to the video based on the base three-dimensional avatar and features of the video, the features comprising image differences of the object between adjacent frames of the video;   wherein the method is implemented using a generation model for three-dimensional avatar videos, and the generation model is trained based on a loss function that includes a plurality of cross-modality loss components including an image-audio loss component, an image-text loss component, and an audio-text loss component; and   wherein the loss function is determined at least in part by:   determining the image-audio loss component as an image-audio contrastive loss function based on image data and audio data;   determining the image-text loss component as an image-text contrastive loss function based on the image data and text data;   determining the audio-text loss component as an audio-text contrastive loss function based on the audio data and the text data; and   determining the loss function based on the image-audio contrastive loss function, the image-text contrastive loss function, and the audio-text contrastive loss function.   
     
     
         2 . The method according to  claim 1 , wherein generating the base three-dimensional avatar comprises:
 generating a three-dimensional projection representation of the object based on shape, posture, and expression of the object in the first frame and the two-dimensional avatar image.   
     
     
         3 . The method according to  claim 2 , wherein generating a three-dimensional projection representation of the object comprises:
 generating the three-dimensional projection representation of the object based on the shape, posture, expression, and texture details of the object in the first frame and the two-dimensional avatar image.   
     
     
         4 . The method according to  claim 3 , wherein generating the base three-dimensional avatar further comprises:
 generating the base three-dimensional avatar based on a camera position at which the first frame was captured, color and lighting information of the two-dimensional avatar image, and the three-dimensional projection representation.   
     
     
         5 . The method according to  claim 1 , wherein the features of the video further comprise voice features and text features, and generating the three-dimensional avatar video comprises:
 acquiring fused features of the video based on features of the image differences, the voice features of the video, and the text features of the video; and   generating the three-dimensional avatar video based on the base three-dimensional avatar and the fused features.   
     
     
         6 . The method according to  claim 1 , wherein the method further comprises:
 generating, based on a first frame of a source video comprising a three-dimensional avatar, audio data of the source video, and text data of the source video, a predictive video comprising the three-dimensional avatar; and   determining the loss function for training the generation model based on image data in the source video and the predictive video, the audio data, and the text data.   
     
     
         7 . The method according to  claim 6 , further comprising training the generation model by using the loss function. 
     
     
         8 . The method according to  claim 6 , further comprising training the generation model by using one or more of the following:
 a motion loss function for determining a motion loss of a three-dimensional avatar video generated by the generation model; and   a style loss function for determining a style loss of the three-dimensional avatar video generated by the generation model.   
     
     
         9 . The method according to  claim 1 , wherein determining the loss function further comprises:
 determining a landmark loss function based on the image data; and   determining the loss function based on the image-audio contrastive loss function, the image-text contrastive loss function, the audio-text contrastive loss function, and the landmark loss function.   
     
     
         10 . A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions, wherein the machine-executable instructions, when executed by a machine, cause the machine to perform actions comprising:
 generating, based on a reference image and a first frame of a video comprising an object, a two-dimensional avatar image of the object;   generating a base three-dimensional avatar of the object by performing a three-dimensional transformation on the two-dimensional avatar image and the object in the first frame; and   generating a three-dimensional avatar video corresponding to the video based on the base three-dimensional avatar and features of the video, the features comprising image differences of the object between adjacent frames of the video;   wherein the actions are implemented using a generation model for three-dimensional avatar videos, and the generation model is trained based on a loss function that includes a plurality of cross-modality loss components including an image-audio loss component, an image-text loss component, and an audio-text loss component; and   wherein the loss function is determined at least in part by:   determining the image-audio loss component as an image-audio contrastive loss function based on image data and audio data;   determining the image-text loss component as an image-text contrastive loss function based on the image data and text data;   determining the audio-text loss component as an audio-text contrastive loss function based on the audio data and the text data; and   determining the loss function based on the image-audio contrastive loss function, the image-text contrastive loss function, and the audio-text contrastive loss function.   
     
     
         11 . An electronic device, comprising:
 at least one processor; and   memory coupled to the at least one processor, wherein the memory has instructions stored therein which, when executed by the at least one processor, cause the electronic device to perform actions comprising:   generating, based on a reference image and a first frame of a video comprising an object, a two-dimensional avatar image of the object;   generating a base three-dimensional avatar of the object by performing a three-dimensional transformation on the two-dimensional avatar image and the object in the first frame; and   generating a three-dimensional avatar video corresponding to the video based on the base three-dimensional avatar and features of the video, the features comprising image differences of the object between adjacent frames of the video;   wherein the actions are implemented using a generation model for three-dimensional avatar videos, and the generation model is trained based on a loss function that includes a plurality of cross-modality loss components including an image-audio loss component, an image-text loss component, and an audio-text loss component; and   wherein the loss function is determined at least in part by:   determining the image-audio loss component as an image-audio contrastive loss function based on image data and audio data;   determining the image-text loss component as an image-text contrastive loss function based on the image data and text data;   determining the audio-text loss component as an audio-text contrastive loss function based on the audio data and the text data; and   determining the loss function based on the image-audio contrastive loss function, the image-text contrastive loss function, and the audio-text contrastive loss function.   
     
     
         12 . The electronic device according to  claim 11 , wherein generating the base three-dimensional avatar comprises:
 generating a three-dimensional projection representation of the object based on shape, posture, and expression of the object in the first frame and the two-dimensional avatar image.   
     
     
         13 . The electronic device according to  claim 12 , wherein generating a three-dimensional projection representation of the object comprises:
 generating the three-dimensional projection representation of the object based on the shape, posture, expression, and texture details of the object in the first frame and the two-dimensional avatar image.   
     
     
         14 . The electronic device according to  claim 13 , wherein generating the base three-dimensional avatar further comprises:
 generating the base three-dimensional avatar based on a camera position at which the first frame was captured, color and lighting information of the two-dimensional avatar image, and the three-dimensional projection representation.   
     
     
         15 . The electronic device according to  claim 11 , wherein the features of the video further comprise voice features and text features, and generating the three-dimensional avatar video comprises:
 acquiring fused features of the video based on features of the image differences, the voice features of the video, and the text features of the video; and   generating the three-dimensional avatar video based on the base three-dimensional avatar and the fused features.   
     
     
         16 . The electronic device according to  claim 11 , wherein the actions further comprise:
 generating, based on a first frame of a source video comprising a three-dimensional avatar, audio data of the source video, and text data of the source video, a predictive video comprising the three-dimensional avatar; and   determining the loss function for training the generation model based on image data in the source video and the predictive video, the audio data, and the text data.   
     
     
         17 . The electronic device according to  claim 16 , wherein the actions further comprise training the generation model by using the loss function. 
     
     
         18 . The electronic device according to  claim 11 , wherein determining the loss function further comprises:
 determining a landmark loss function based on the image data; and   determining the loss function based on the image-audio contrastive loss function, the image-text contrastive loss function, the audio-text contrastive loss function, and the landmark loss function.