Method and Apparatus for Processing Reference Inputs for Video Compositing with Replacement

Abstract

A compositor generates a representation of a gradient image corresponding to a replaceable background object in a scene. The representation can be generated from user set of reference points in the scene.

1 . A computer-implemented method for generating multigradient image components in an image processing system, the method comprising:
 under control of one or more computer systems configured with executable instructions:
 obtaining a first indication of a set of reference pixels, each having a pixel location in an image array of a first image; 
 obtaining a second indication comprising reference pixel color values associated with each reference pixel of the set of reference pixels; 
 computing a gradient array from the reference pixel values and reference pixel locations, wherein the gradient array represents gradient values for pixels over the image array based on reference pixel values and the reference pixel locations wherein the gradient values correspond to interpolated values over the gradient array and align with the reference pixel values at their respective reference pixel locations; and 
 processing the first image using the gradient array. 
   
     
     
         2 . (canceled) 
     
     
         3 . The computer-implemented method of  claim 1 , further comprising:
 obtaining at least one image, including the first image, of live action using an image capture device or system;   generating a displayable representation of the first image, wherein the set of reference pixels comprises first image pixels of the first image; and   determining the reference pixel values and the reference pixel locations for each reference pixel of the set of reference pixels,   wherein processing the image comprises processing a second image using the color gradient array.   
     
     
         4 . The computer-implemented method of  claim 3 , further comprising:
 presenting the displayable representation to a user on an image display device; and   accepting inputs from the user selecting the set of reference pixels on the image display device.   
     
     
         5 . The computer-implemented method of  claim 1 , wherein the set of reference pixels are pixels selected to correspond with pixels representing a background element of a scene that is to be replaced with corresponding pixels or subpixels of a background image. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein computing the gradient array comprises interpolating values at interpolation locations other than the reference pixel locations by computing a radial basis function from the reference pixel values and the reference pixel locations and setting gradient values for interpolation pixels other than the reference pixels of the set of reference pixels according to the radial basis function, wherein the radial basis function is a thin-plate spline radial basis function. 
     
     
         7 - 8 . (canceled) 
     
     
         9 . The computer-implemented method of  claim 1 , wherein a specific reference pixel value associated with a specific reference pixel is a function of multiple pixels of a captured image. 
     
     
         10 . The computer-implemented method of  claim 9 , wherein the multiple pixels are pixels in a window of the captured image relative to the specific reference pixel and the function comprises a Gaussian blob function of pixels in the window. 
     
     
         11 . The computer-implemented method of  claim 9 , wherein the multiple pixels are pixels in a window of the captured image relative to the specific reference pixel and the function comprises a uniform weighting of the pixels in the window and the window is a rectangular pixel array. 
     
     
         12 . The computer-implemented method of  claim 1 , wherein computing the gradient array comprises:
 determining a first proportion for an interpolation pixel at an interpolation location other than one of the reference pixel locations, wherein the first proportion represents a proportion of a foreground contribution to a pixel value of the interpolation pixel and a background contribution to the pixel value of the interpolation pixel; and   representing an interpolating pixel value at the interpolation location based, in part, on the first proportion.   
     
     
         13 . The computer-implemented method of  claim 12 , further comprising representing the first proportion for the interpolation pixel and first proportions for other interpolation pixels in an alpha channel. 
     
     
         14 . The computer-implemented method of  claim 1 , further comprising:
 processing the first image using the gradient array to determine background pixels, foreground pixels, and mixed foreground-background pixels;   subtracting a first set of gradient array pixel values from the background pixels;   partially subtracting a second set of gradient array pixel values from the mixed foreground-background pixels; and   merging a resulting image with a replacement background image.   
     
     
         15 . The computer-implemented method of  claim 14 , further comprising determining a first proportion for an interpolation pixel at an interpolation location other than one of the reference pixel locations, wherein the first proportion represents a proportion of a foreground contribution to a of the interpolation pixel and a background contribution to the of the interpolation pixel, wherein partially subtracting comprises subtracting based, in part, on the first proportion. 
     
     
         16 . The computer-implemented method of  claim 1 , further comprising:
 merging the gradient array with a second image to form a third image, wherein in the gradient array contributes to pixel colors in the third image.   
     
     
         17 . The computer-implemented method of  claim 1 , further comprising:
 obtaining a third image, the third image representing a live action scene;   determining a time relationship between the first image and the third image;   determining a set of correspondence pixels in the third image that correspond to the reference pixels of the set of reference pixels of the first image by matching pixels of the third image with pixels in the first image based on respective pixel values;   determining a second set of reference values and a second set of pixel locations for correspondence pixels in the set of reference pixels;   computing a second gradient array, for the third image, from the second set of reference values and the second set of pixel locations of the first image, wherein the second gradient array represents gradient values for pixels over the image array based on the third image; and   processing the third image into a fourth image using the second gradient array.   
     
     
         18 . (canceled) 
     
     
         19 . The computer-implemented method of  claim 17 , further comprising:
 determining movement of scene elements in the third image relative to the first image; and   selecting the correspondence pixels in the third image based on pixel positions of the reference pixels of the first image and determined movement of the scene elements, to at least approximately align the correspondence pixels with objects in a scene.   
     
     
         20 . The computer-implemented method of  claim 19 , wherein determining the movement of the scene elements in the third image relative to the first image comprises:
 identifying a first set of tracking markers in the first image;   selecting the reference pixels relative to the first set of tracking markers;   identifying a second set of tracking markers in the third image, the second set of tracking markers corresponding to the first set of tracking markers; and   selecting the correspondence pixels in the third image based on the second set of tracking markers and relative positions of the reference pixels relative to the first set of tracking markers.   
     
     
         21 . A non-transitory computer-readable storage medium storing instructions, which when executed by at least one processor of a computer system, causes the computer system to carry out the method of  claim 1 . 
     
     
         22 - 23 . (canceled) 
     
     
         24 . A non-transitory carrier medium carrying image data that includes a composited image and/or a gradient image generated according to the method of  claim 1 . 
     
     
         25 . The computer-implemented method of  claim 1 , wherein a pixel value, of the reference pixel values, is one or more of: a color value, a depth value, an alpha value, a weight value, and/or an object identifier value. 
     
     
         26 . An image processor comprising:
 a first memory storage for images stored in computer-readable form;   a frame selector for selecting a first image from the first memory storage, wherein content of the first image is represented by an image array;   a user interface, coupled to the frame selector, for presenting the first image and obtaining a first indication of a set of reference pixels, each reference pixel having a pixel location in the image array;   a compositor coupled to the user interface, including second memory storage for a second indication comprising at least an indication of a set of reference pixel values associated reference pixels of the set of reference pixels;   image storage, accessible by the compositor, for storing a computed gradient array computed from the set of reference pixel values and reference pixel locations, wherein the computed gradient array represents gradient values for pixels over the image array based on reference pixel values and the reference pixel locations wherein the gradient values correspond to interpolated values over the computed gradient array and align with the set of reference pixel values at their respective reference pixel locations; and   program code for the first image using the computed gradient array.