Refining motion vectors in video motion estimation

Abstract

In video motion estimation an initial candidate motion vector is generated for each block and a vector error is determined as for example a DFD. Spatial gradients of pixel values are calculated and used to refine the initial candidate motion vector. The relative contribution of the spatial gradients to the refinement process depends on the vector error.

1 . A video motion estimation method, comprising the steps in a processor of
 dividing a frame into regions;   generating for each region at least one initial candidate motion vector to represent the respective positions of the block's pixels in a respective reference frame, the or each initial candidate motion vector being common to all pixels in the frame;   determining for each pixel a vector error representing the error in using that initial candidate motion vector to represent the position of that pixel in the reference frame;   determining spatial gradients of pixel values;   refining each initial candidate motion vector through use of said spatial gradients of pixel values to derive one or more refined motion vectors the relative contribution of spatial gradients of pixels to the refinement process depending on said vector error; and   assigning one refined motion vector to each pixel.   
     
     
         2 . A method according to  claim 1  in which the contribution of pixels to the refinement process is determined by multiplication by a weight function which is a decreasing function of the vector error. 
     
     
         3 . A method according to  claim 2  in which the weight function is a negative exponential function of the vector error. 
     
     
         4 . A method according to  claim 1 , in which a first spatial gradient is determined at the location of the pixel in the frame and a second spatial gradient is determined at the location in the reference frame for that pixel as represented by the initial candidate motion vector. 
     
     
         5 . A method according to  claim 1  in which a motion vector assigned to a pixel in a region in a current frame is refined in dependence upon a combination of a plurality of sets of pixel-value gradient parameters for a respective plurality of pixels at locations in the said region of the current frame. 
     
     
         6 . A method according to  claim 1  in which a motion vector assigned to a pixel in a region in a current frame is refined in dependence upon a combination of a plurality of sets of pixel-value gradient parameters for a respective plurality of pixels at locations in the said reference frame defined by motion vectors assigned to pixels in the said region of the current frame. 
     
     
         7 . A method according to  claim 6  in which, for each pixel and for each vector field, horizontal and vertical gradients of a current frame and of a reference frame displaced according to the vector field are taken, sums of products of the gradients, of motion vector components and displaced frame differences are taken to form autocorrelation and cross-correlation signals, those signals are multiplied by weights and filtered by spatial smoothing filters, and the resulting smoothed weighted correlation signals are combined by taking quotients of sums of products to produce refined motion vectors. 
     
     
         8 . A method according to  claim 1  in which initial candidate motion vectors are derived from motion estimation applied to overlapping rectangular image blocks, with an overlap structure such that each pixel not close to the picture edges occupies four regions; and, an assignment process allocates refined vectors to pixels in dependence on respective error values for the refined vectors. 
     
     
         9 . Video motion estimation apparatus comprising:
 a motion estimator for receiving two or more video frames and generating initial candidate vectors;   a displaced frame difference unit for measuring a vector error for each initial candidate vector;   a vector refiner which receives at least one video frame, the initial candidate vectors and the vector errors and which in a refinement process outputs candidate vectors; and   a vector assignment unit which in an assignment process assigns one of the candidate vectors to pixels of a video frame;   wherein the vector refiner makes use of spatial gradients of pixel values, and the relative contribution of spatial gradients of pixel values to the refinement process depends on said vector errors.   
     
     
         10 . Apparatus according to  claim 9  in which the contribution of pixels to the refinement process is determined by multiplication by a weight function which is a decreasing function of the vector error. 
     
     
         11 . Apparatus according to  claim 10  in which the function is a negative exponential function. 
     
     
         12 . Apparatus according to  claim 9  in which a motion vector assigned to a pixel in a region in a current frame is refined in dependence upon a combination of a plurality of sets of pixel-value gradient parameters for a respective plurality of pixels at locations in the said region of the current frame. 
     
     
         13 . Apparatus according to  claim 9  in which a motion vector assigned to a pixel in a region in a current frame is refined in dependence upon a combination of a plurality of sets of pixel-value gradient parameters for a respective plurality of pixels at locations in the said reference frame defined by motion vectors assigned to pixels in the said region of the current frame. 
     
     
         14 . Apparatus according to  claim 13  in which, for each pixel and for each vector field, horizontal and vertical gradients of a current frame and of a reference frame displaced according to the vector field are taken, sums of products of the gradients, of motion vector components and displaced frame differences are taken to form autocorrelation and cross-correlation signals, those signals are multiplied by weights and filtered by spatial smoothing filters, and the resulting smoothed weighted correlation signals are combined by taking quotients of sums of products to produce refined motion vectors. 
     
     
         15 . Apparatus according to  claim 9  in which initial candidate motion vectors are derived from motion estimation applied to overlapping rectangular image blocks, with an overlap structure such that each pixel not close to the picture edges occupies four regions; and, an assignment process allocates refined vectors to pixels in dependence on respective error values for the refined vectors. 
     
     
         16 . In a video motion estimation system, a method for generating candidate motion vector fields that represent the respective positions of a frame's pixels in a respective reference frame, comprising in a processor the steps of:
 forming initial candidate vectors each having a respective vector error;   forming spatial gradients of pixel values; and   refining said initial candidate vectors, in a refinement process to form candidate vectors for a subsequent vector assignment process;   in which the relative contribution of said spatial gradients of pixel values to the refinement process depends on a said vector error.   
     
     
         17 . A method according to  claim 16  in which the contribution of pixels to the refinement process is determined by multiplication by a weight function which is a decreasing function of the vector error. 
     
     
         18 . A method according to  claim 17  in which the function is a negative exponential function. 
     
     
         19 . A method according to  claim 16  in which a motion vector assigned to a pixel in a region in a current frame is refined in dependence upon a combination of a plurality of sets of pixel-value gradient parameters for a respective plurality of pixels at locations in the said region of the current frame. 
     
     
         20 . A method according to  claim 16  in which a motion vector assigned to a pixel in a region in a current frame is refined in dependence upon a combination of a plurality of sets of pixel-value gradient parameters for a respective plurality of pixels at locations in the said reference frame defined by motion vectors assigned to pixels in the said region of the current frame.