Methods, systems, and computer program products for creating three-dimensional video sequences

Abstract

Methods and systems for creating three-dimensional video sequences of a scene are disclosed. An example method can include receiving multiple frames of a scene. The method may include selecting a target frame from among the multiple frames; selecting a first subset of frames, N, from among the multiple frames that are associated with the target frame that is representative of a large stereo baseline; and analyzing the first frame subset to identify two images for forming a stereoscopic pair of frames. Further, the method includes extracting depth data of static objects in the stereoscopic pair. The method includes selecting a second subset of frames that are associated with the target frame that is representative of a smaller stereo baseline than that represented by N; and utilizing the second frame subset to calculate depth of moving objects. The method includes generating a three-dimensional video frame based on the depth data.

What is claimed: 
     
         1 . A method for creating a three-dimensional video sequence of a scene, the method comprising:
 using at least one processor and memory for:   receiving a two-dimensional video sequence of a scene, the two-dimensional video sequence including a plurality of frames;   selecting a target frame from among the plurality of frames;   selecting a first subset of frames, N, from among the plurality of frames that are associated with the target frame, representative of a targeted stereo displacement of the capture device from the target frame position;   analyzing the first subset of frames to identify two images for use in forming a stereoscopic pair of frames with a predetermined spatial difference;   identifying and classifying static and moving objects between the stereoscopic pair of frames;   extracting depth data of static objects in the stereoscopic pair of frames;   selecting a second subset of frames, n, from among the plurality of frames that are associated with the target frame, representative of a stereo displacement that is substantially smaller than the displacement represented by N;   utilizing the second subset of frames to calculate depth of moving objects;   combining the depth values of static and moving objects based on an absolute depth of static objects and the relative depth of moving objects; and   generating a three-dimensional video frame consisting of the target frame and a depth-generated view corresponding to the target frame based on the depth data.   
     
     
         2 . The method of  claim 1 , further comprising one or more steps of identifying suitable frames, registration, stabilization, color correction, transformation, and depth adjustment. 
     
     
         3 . The method of  claim 1 , further comprising generating one or more additional frames and frame viewpoints using one of existing raster data and depth information. 
     
     
         4 . The method of  claim 1 , further comprising using a micro stereo base technique for generating image representations of close and moving objects of the scene. 
     
     
         5 . The method of  claim 1 , further comprising using a macro stereo base technique for generating image representations of a background and non-moving objects of the scene. 
     
     
         6 . The method of  claim 1 , further comprising using an image capture device for capturing the two-dimensional video sequence. 
     
     
         7 . The method of  claim 1 , further comprising using the steps of  claim 1  for generating a plurality of three-dimensional video frames. 
     
     
         8 . The method of  claim 1 , further comprising dividing the captured two-dimensional video sequence into segments divided by utilizing one of a scene change detection and camera pose information. 
     
     
         9 . The method of  claim 8 , further comprising creating a stereoscopic video sequence from each segment. 
     
     
         10 . The method of  claim 9 , further comprising equalizing the depths and other three-dimensional parameters and combining the individual stereoscopic segments to form a single three-dimensional video stream. 
     
     
         11 . The method of  claim 1 , further comprising analyzing captured frames to measure object displacements between two or more frames via motion vectors and identifying and classifying static and moving objects within the current target frame. 
     
     
         12 . The method of  claim 1 , further comprising:
 analyzing captured frames; and   identifying a position of key static objects to estimate the position of the camera.   
     
     
         13 . The method of  claim 1 , further comprising measuring the motion vectors of the moving objects and estimating their relative position on the x, y space. 
     
     
         14 . The method of  claim 1 , further comprising:
 measuring sizes of moving objects; and   estimating their relative position on the z-space based on the rate of increase or decrease of their sizes.   
     
     
         15 . The method of  claim 1 , further comprising estimating the depth of moving objects by identifying the depth of key points of moving objects relative to the depth of neighboring static objects with known depths and extrapolating based on temporal or structural relationships. 
     
     
         16 . The method of  claim 1 , further comprising estimating the depth of moving objects by determining whether they are moving behind or in front of static objects and utilizing the depth of static objects to estimate the depth of a moving object based on the moving object's trajectory in time. 
     
     
         17 . The method of  claim 1 , further comprising:
 segmenting the static portions of the scene into static objects with known positions in three-dimensional space and utilizing prediction techniques to estimate the depth of new information entering into the captured video sequence associated with the same or related static objects.   
     
     
         18 . The method of  claim 1 , further comprising:
 measuring sizes of static objects; and   estimating changes in camera focal length and adjusting the depth of the scene based on the relative zoom factor of the camera based on their relative rate of increase or decrease over time.   
     
     
         19 . A system for creating a three-dimensional video sequence of a scene, the system comprising:
 a memory having stored therein computer program code;   a computer processor that executes the computer program code;   a video generator configured to:
 receive a two-dimensional video sequence of a scene, the two-dimensional video sequence including a plurality of frames; 
 select a target frame from among the plurality of frames; 
 select a first subset of frames, N, from among the plurality of frames that are associated with the target frame, representative of a targeted stereo displacement of the capture device from the target frame position; 
 analyze the first subset of frames to identify two images for use in forming a stereoscopic pair of frames with a predetermined spatial difference; 
 extract depth data of static objects in the stereoscopic pair of frames; 
 select a second subset of frames, n, from among the plurality of frames that are associated with the target frame, representative of a stereo displacement that is substantially smaller than the displacement represented by N; 
 utilize the second subset of frames to calculate depth of moving objects; 
 combine the depth values of static and moving objects based on the absolute depth of static objects and the relative depth of moving objects; and 
 generate a three-dimensional video frame consisting of the target frame and a depth-generated view corresponding to the target frame based on the depth data. 
   
     
     
         20 . The system of  claim 19 , wherein the video generator is configured to implement one or more functions of identifying suitable frames, registration, stabilization, color correction, transformation, and depth adjustment.