Method and device for video image processing, calculating the similarity between video frames, and acquiring a synthesized frame by synthesizing a plurality of contiguous sampled frames

Abstract

To acquire a high-resolution frame from a plurality of frames sampled from a video image, it is necessary to obtain a high-resolution frame with reduced picture quality degradation regardless of motion of a subject included in the frame. Because of this, between a plurality of contiguous frames Fr N and Fr N+1 , there is estimated a correspondent relationship. Based on the correspondent relationship, the frames Fr N+1 and Fr N are interposed to obtain first and second interpolated frames Fr H1 and Fr H2 . Based on the correspondent relationship, the coordinates of the frame Fr N+1 are transformed, and from a correlation value with the frame Fr N , there is obtained a weighting coefficient α(x o , y o ) that makes the weight of the first interpolated frame Fr H1 greater as a correlation becomes greater. With the weighting coefficient, the first and second interpolated frames are weighted and added to acquire a synthesized frame Fr G .

1 . A video image synthesis method comprising the steps of:
 sampling two contiguous frames from a video image;   placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relation between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;   acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said corresponding relation;   acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;   acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of said other frame;   acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and   acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient.   
     
     
         2 . A video image synthesis method comprising the steps of:
 sampling three or more contiguous frames from a video image;   placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating corresponding relations between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;   acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said corresponding relations;   acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;   acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of each of said other frames;   acquiring weighting coefficients that make a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and   acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.   
     
     
         3 . The synthesis method as set forth in  claim 1 , wherein
 when said edge information has been computed for each of the pixels that constitute each of said frames,   said weighting coefficient is interpolated to acquire weighting coefficients for all pixels that constitute said first and second interpolated frames.   
     
     
         4 . The synthesis method as set forth in  claim 2 , wherein
 when said edge information has been computed for each of the pixels that constitute each of said frames,   said weighting coefficient is interpolated to acquire weighting coefficients for all pixels that constitute said first and second interpolated frames.   
     
     
         5 . The synthesis method as set forth in  claim 1 , wherein the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame are performed by employing at least one component that constitutes said frame. 
     
     
         6 . The synthesis method as set forth in  claim 2 , wherein the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame are performed by employing at least one component that constitutes said frame. 
     
     
         7 . A video image synthesizer comprising:
 sampling means for sampling two contiguous frames from a video image;   correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;   first interpolation means for acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relation;   second interpolation means for acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;   edge information acquisition means for acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of said other frame;   weighting-coefficient acquisition means for acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and   synthesis means for acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient.   
     
     
         8 . A video image synthesizer comprising:
 sampling means for sampling three or more contiguous frames from a video image;   correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;   first interpolation means for acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;   second interpolation means for acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;   edge information acquisition means for acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of each of said other frames;   weighting-coefficient acquisition means for acquiring weighting coefficients that make a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and   synthesis means for acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.   
     
     
         9 . The video image synthesizer as set forth in  claim 7 , wherein
 when said edge information has been computed for each of the pixels that constitute each of said frames,   said weighting-coefficient acquisition means performs interpolation on said weighting coefficient, thereby acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.   
     
     
         10 . The video image synthesizer as set forth in  claim 8 , wherein
 when said edge information has been computed for each of the pixels that constitute each of said frames,   said weighting-coefficient acquisition means performs interpolation on said weighting coefficients, thereby acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.   
     
     
         11 . The video image synthesizer as set forth in  claim 7 , wherein said correspondent relationship estimation means, said first interpolation means, said second interpolation means, said edge information acquisition means, said weighting-coefficient acquisition means, and said synthesis means perform the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame, by employing at least one component that constitutes said frame. 
     
     
         12 . The video image synthesizer asset forth in  claim 8 , wherein said correspondent relationship estimation means, said first interpolation means, said second interpolation means, said edge information acquisition means, said weighting-coefficient acquisition means, and said synthesis means perform the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame, by employing at least one component that constitutes said frame. 
     
     
         13 . A program for causing a computer to execute a video image synthesis method comprising:
 a procedure of sampling two contiguous frames from a video image;   a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;   a procedure of acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;   a procedure of acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;   a procedure of acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of said other frame;   a procedure of acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and   a procedure of acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient.   
     
     
         14 . A program for causing a computer to execute a video image synthesis method comprising:
 a procedure of sampling three or more contiguous frames from a video image;   a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;   a procedure of acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;   a procedure of acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;   a procedure of acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of each of said other frames;   a procedure of acquiring weighting coefficients that make a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and   a procedure of acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.   
     
     
         15 . The program as set forth in  claim 13 , wherein
 when said edge information has been computed for each of the pixels that constitute each of said frames,   said weighting-coefficient acquisition procedure is a procedure of performing interpolation on said weighting coefficient and acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.   
     
     
         16 . The program as set forth in  claim 14 , wherein
 when said edge information has been computed for each of the pixels that constitute each of said frames,   said weighting-coefficient acquisition procedure is a procedure of performing interpolation on said weighting coefficients and acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.   
     
     
         17 . The program as set forth in  claim 13 , wherein said correspondent relationship estimation procedure, said first interpolated frame acquisition procedure, said second interpolated frame acquisition procedure, said edge information acquisition procedure, said weighting-coefficient acquisition procedure, and said synthesized-frame acquisition procedure are procedures of performing the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame by employing at least one component that constitutes said frame, respectively. 
     
     
         18 . The program as set forth in  claim 14 , wherein said correspondent relationship estimation procedure, said first interpolated frame acquisition procedure, said second interpolated frame acquisition procedure, said edge information acquisition procedure, said weighting-coefficient acquisition procedure, and said synthesized-frame acquisition procedure are procedures of performing the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame by employing at least one component that constitutes said frame, respectively.