Method for editing performers in video using virtual human

Abstract

A method for editing performers in a video using a virtual human includes: (a) selecting frames including a source human object by searching for a 2D video composed of a plurality of frames; (b) generating a face sequence by tracking a face of the source human object; (c) finding a depth of each object, and determining whether a corresponding object is located in front of or rear of the source human object; (d) selecting a 3D face template, and generating a 3D face of a target human object by reflecting a 2D face of the target human object; (f) estimating pose information, generating a 3D virtual human of the target human object by reflecting the pose information, and synthesizing the 3D face of the target human object with a face portion of the 3D virtual human; and (i) synthesizing the 3D virtual human into a 2D video image space.

What is claimed is: 
     
         1 . A method for editing performers in a video using a virtual human, the method comprising:
 (a) selecting frames including a person of a human object to be edited (hereinafter referred to as a source human object) by searching for a 2D video composed of a plurality of consecutive frames in frame units;   (b) generating a face sequence of the source human object by tracking a face of the source human object in the selected frames;   (c) finding a depth of each object existing in a corresponding frame by generating a depth map, and determining whether a corresponding object is located in front of or rear of the source human object;   (d) selecting a 3D face template for the face of the source human object, and generating a 3D face of a target human object by reflecting a 2D face of the target human object in the selected 3D face template;   (f) estimating pose information from the source human object for each of the selected frames, generating a 3D virtual human of the target human object by reflecting the pose information in a preset 3D body template, and synthesizing the 3D face of the target human object with a face portion of the 3D virtual human of the target human object; and   (i) synthesizing the 3D virtual human of the target human object into a 2D video image space of the corresponding frame.   
     
     
         2 . The method of  claim 1 , wherein, in step (c), the object separated according to the depth is composed of a background object located at a rearmost side, a source human object that is an editing target, and a general object that is neither the background object nor the source human object, and
 in step (i), each separated object is synthesized into the 2D video space, in which the separated object is sequentially synthesized into the 2D video space from an object that is located at the rearmost side according to a front-rear positional relationship, and when the separated object is the general object, the general object is synthesized into the 2D video space as it is, and when the separated object is the source human object, a virtual human of a target human object corresponding to the source human object is synthesized into the 2D video space.   
     
     
         3 . The method of  claim 2 , wherein in step (c), all general objects located rear of a human object that is located at the rearmost side among source human objects of the editing target are integrated into the background object. 
     
     
         4 . The method of  claim 1 , further comprising (h) correcting a separated background object in association with a region of the separated object,
 wherein, in step (i), when each separated object is synthesized into the 2D video space, the corrected background is synthesized instead of the separated background object, and is first synthesized into the 2D video space.   
     
     
         5 . The method of  claim 1 , wherein, in step (d), a 3D face template corresponding to the face of the source human object is selected, a 3D face of the target human object is generated by reflecting the 2D face of the target human object in the 3D face template, and a position and a direction of the 3D face of the target human object are synchronized with a size, a position, and a direction of the 2D face of the source human object. 
     
     
         6 . The method of  claim 5 , wherein, in step (d), the 3D face of the target human object and the 2D face of the source human object are synchronized to coincide with each other in terms of the position and the direction using landmarks including at least one of an area around eyes, an area around a mouth, a jawline, and a nose line. 
     
     
         7 . The method of  claim 1 , further comprising (e) estimating lighting information including a light source position and a lighting intensity with respect to each of the selected frames,
 wherein, in step (i), an outer appearance of the 3D virtual human of the target human object is rendered by reflecting the estimated lighting information, and the rendered 3D virtual human is synthesized by projecting the rendered 3D virtual human onto a 2D plane.   
     
     
         8 . The method of  claim 1 , wherein, in step (f), the 3D virtual human of the target human object is generated from a 2D image of a corresponding frame of the 2D video, in which the 3D virtual human of the target human object is generated using a skinned multi-person linear model-eXpretive (SMPL-X) deep learning model, and pose information and an outer appearance of the 3D virtual human are extracted. 
     
     
         9 . The method of  claim 1 , further comprising (g) animating a 3D virtual human of the target human object according to a preset pose sequence for sections of frames k+1 to k+n to generate a 3D virtual human of each frame by rigging the 3D virtual human of a frame k and reflecting and the pose sequence in the rigged 3D virtual human,
 wherein, in step (i), the 3D virtual human generated by animating in step (g) is synthesized into a 2D video space.   
     
     
         10 . A computer-readable recording medium having a program recorded thereon for executing the method for editing the performers in the video using the virtual human of  claim 1 . 
     
     
         11 . A computer-readable recording medium having a program recorded thereon for executing the method for editing the performers in the video using the virtual human of  claim 2 . 
     
     
         12 . A computer-readable recording medium having a program recorded thereon for executing the method for editing the performers in the video using the virtual human of  claim 3 . 
     
     
         13 . A computer-readable recording medium having a program recorded thereon for executing the method for editing the performers in the video using the virtual human of  claim 4 . 
     
     
         14 . A computer-readable recording medium having a program recorded thereon for executing the method for editing the performers in the video using the virtual human of  claim 5 . 
     
     
         15 . A computer-readable recording medium having a program recorded thereon for executing the method for editing the performers in the video using the virtual human of  claim 6 . 
     
     
         16 . A computer-readable recording medium having a program recorded thereon for executing the method for editing the performers in the video using the virtual human of  claim 7 . 
     
     
         17 . A computer-readable recording medium having a program recorded thereon for executing the method for editing the performers in the video using the virtual human of  claim 8 . 
     
     
         18 . A computer-readable recording medium having a program recorded thereon for executing the method for editing the performers in the video using the virtual human of  claim 9 .