Automatic tracking matte system

Abstract

A system for generating automatically tracking mattes that rapidly integrates live action and virtual composite images.

1 . A method comprising:
 generating an outline around an object, which is to be removed or preserved, in a 2D display of a live action image from a live action camera;   converting the shape defined by the 2D outline into a 3D mesh, wherein the converting includes removing lens distortion;   projecting the 3D mesh from a virtual camera having the same position and orientation as the live action camera;   creating a 2D undistorted matte shape from the 3D mesh;   distorting the 2D undistorted matte shape to match the as-shot, distorted live action image; and   processing, using the 2D distorted matte shape, portions of the live action image with other images to form a composite image.   
     
     
         2 . A method comprising:
 generating an outline around an object, which is to be removed or preserved, in a 2D display of a live action image from a live action camera;   converting the shape defined by the 2D outline into a 3D mesh;   projecting the 3D mesh from a virtual camera having the same position and orientation as the live action camera;   creating a 2D undistorted matte shape from the 3D mesh; and   processing, using the 2D undistorted matte shape, portions of the live action image with other images to form a composite image.   
     
     
         3 . The method of  claim 2  wherein the converting includes removing lens distortion. 
     
     
         4 . The method of  claim 2  wherein the processing includes distorting the 2D undistorted matte shape to match the as-shot, distorted live action image. 
     
     
         5 . The method of  claim 2  wherein the user interface automatically handles distortion processing of points of the user selected outline points. 
     
     
         6 . The method of  claim 2  wherein the normal of the direction the camera was facing when the geometry was first created is preserved in the system, so that even after the camera moves, the matte size and 3D position can be adjusted while preserving the correct match of the outline to the live action object. 
     
     
         7 . The method of  claim 2  wherein the outline generation step includes editing the 3D points by dragging them around in a 2D interface, while preserving their location on their original 2D plane. 
     
     
         8 . The method of  claim 2  wherein the method includes real time undistortion and re-distortion, to thereby create correctly matched geometry. 
     
     
         9 . A video processing system, comprising:
 a distortion removal processor programmed to create a set of undistorted points;   a 2D-to-3D converter configured to use the set of undistorted points to calculate 3D matte geometry;   a 3D renderer configured to use the 3D matte geometry to generate a 2D undistorted matte shape;   a distortion addition processor programmed to use the 2D undistorted matte shape to create a distorted 2D matte shape; and   a 2D compositor configured to use the distorted 2D matte shape to combine at least one portion of a live action image with at least one other image to generate a composited image.   
     
     
         10 . The video processing system of  claim 9  wherein the distortion removal processor uses camera and lens data to create the set of undistorted points. 
     
     
         11 . The video processing system of  claim 9  wherein the 2D-to-3D converter uses lens data to create the distorted 2D matte shape. 
     
     
         12 . The video processing system of  claim 9  wherein the 3D renderer uses camera and lens data to generate the undistorted matte shape. 
     
     
         13 . The video processing system of  claim 9  wherein the distortion removal processor, the 3D renderer and the 2D-to-3D converter are all configured to receive combined data combined from camera data from a camera tracking system and lens data from a lens calibration table. 
     
     
         14 . The video processing system of  claim 9  wherein the 3D renderer is configured to receive background scene geometry and to generate an undistorted background image, and wherein the distortion addition processor is programmed to receive the undistorted background image and to generate a background image for delivery to the 2D compositor. 
     
     
         15 . (canceled) 
     
     
         16 . (canceled) 
     
     
         17 . (canceled) 
     
     
         18 . (canceled) 
     
     
         19 . (canceled) 
     
     
         20 . (canceled)