Free viewpoint video

Abstract

A method of remotely viewing a video from a selected viewpoint selected by the viewer from a continuous segment, including, receiving a recording of a video of a subject recorded using a first depth video camera that records a video comprising a sequence of picture frames and additionally records a depth value for pixels of the picture frames; receiving a recording of a video of the subject recorded using a standard video camera or a second depth video camera positioned to record a video at a viewpoint that differs from the viewpoint of the depth video camera; using the recordings to render a viewable video from the selected viewpoint; and display the rendered viewable video to the viewer.

1 . A method of remotely viewing a video from a selected viewpoint selected by the viewer from a continuous segment, comprising:
 receiving a recording of a video of a subject recorded using a first depth video camera that records a video comprising a sequence of picture frames and additionally records a depth value for pixels of the picture frames;   receiving a recording of a video of the subject recorded using a standard video camera or a second depth video camera positioned to record a video at a viewpoint that differs from the viewpoint of the depth video camera;   rendering a depth hull that defines a three dimensional outline of the subject being recorded using the depth values recorded by the depth video cameras;   providing the recorded video from one or more cameras positioned on either side of the selected viewpoint;   incorporating the recorded video from the one or more cameras onto the rendered depth hull to render a viewable video from the selected viewpoint; and   displaying the rendered viewable video to the viewer.   
     
     
         2 . A method according to  claim 1 , wherein said continuous segment encompasses 360 degrees around the subject. 
     
     
         3 . A method according to  claim 1 , wherein said continuous segment encompasses less than 360 degrees around the subject. 
     
     
         4 . A method according to  claim 1 , wherein said rendering is performed using the depth values from at least two depth video cameras. 
     
     
         5 . A method according to  claim 4 , wherein said at least two depth video cameras are positioned on opposite sides of the subject. 
     
     
         6 . A method according to  claim 1 , wherein said rendering is performed using at least two depth video cameras and the recordings of at least two standard video cameras surrounding the subject are incorporated to render the viewable video. 
     
     
         7 . A method according to  claim 1 , wherein the video cameras are operated synchronously. 
     
     
         8 . A method according to  claim 1 , wherein said rendering is performed by a server connected to the video cameras and provided to the remote viewer. 
     
     
         9 . A method according to  claim 1 , wherein said rendering is performed by the remote viewer. 
     
     
         10 . A method according to  claim 1 , wherein said providing comprises transmitting the information to the remote viewers. 
     
     
         11 . A method according to  claim 1 , wherein said incorporating is performed by the renderer of the depth hull. 
     
     
         12 . A method according to  claim 1 , wherein said three dimensional outline surrounds the entire shape of the subject. 
     
     
         13 . A non-transitory computer readable medium comprising a set of computer executable instructions adapted to perform the method of  claim 1 . 
     
     
         14 . A system for remotely viewing a video from a selected viewpoint selected by the viewer from a continuous segment, comprising:
 a first depth video camera that records a video of a subject comprising a sequence of picture frames and additionally records a depth value for pixels of the picture frames;   a standard video camera or a second depth video camera positioned to record a video of the subject at a viewpoint that differs from the viewpoint of the first depth video camera;   a computer adapted to receive the recordings of the video cameras and render a depth hull that defines a three dimensional outline of the subject being recorded using the depth values recorded by the depth video cameras;   one or more remote viewer stations that are adapted to:
 1) select a viewpoint for viewing the recorded subject; 
 2) receive the rendered depth hull; 
 3) receive the recorded video from the recordings of one or more video cameras positioned on either side of the selected viewpoint of the viewer; 
 4) incorporate the received recorded video onto the depth hull to render a viewable video from the selected viewpoint; and 
 5) display the rendered viewable video to the viewer. 
   
     
     
         15 . A system according to  claim 14 , wherein said first depth video camera and said second depth video camera are positioned on opposite sides of the recorded subject. 
     
     
         16 . A system according to  claim 14 , comprising at least two depth video cameras and at least two standard video cameras surrounding the subject. 
     
     
         17 . A system according to  claim 14 , wherein the video cameras are operated synchronously. 
     
     
         18 . A system according to  claim 14 , wherein said computer is adapted to control the order of recording picture frames and depth values by the video cameras. 
     
     
         19 . A system according to  claim 14 , wherein said continuous segment encompasses 360 degrees around the subject. 
     
     
         20 . A system according to  claim 14 , wherein said continuous segment encompasses less than 360 degrees around the subject. 
     
     
         21 . A system according to  claim 14 , wherein said three dimensional outline surrounds the entire shape of the subject. 
     
     
         22 . A system for remotely viewing a video from a selected viewpoint selected by the viewer from a continuous segment, comprising:
 a first depth video camera that records a video of a subject comprising a sequence of picture frames and additionally records a depth value for pixels of the picture frames;   a standard video camera or a second depth video camera positioned to record a video of the subject at a viewpoint that differs from the viewpoint of the first depth video camera;   a computer adapted to receive the recordings of all the video cameras;   one or more remote viewer stations that are adapted to:
 1) render a depth hull that defines a three dimensional outline of the subject being recorded using the depth values recorded by the depth video cameras; 
 2) select a viewpoint for viewing the recorded subject; 
 3) receive the recorded video from the recordings of one or more video cameras positioned on either side of the selected viewpoint of the viewer; 
 4) incorporate the received recorded video onto the depth hull to render a viewable video from the selected viewpoint; and 
 5) display the rendered viewable video to the viewer. 
   
     
     
         23 . A system according to  claim 22 , wherein said three dimensional outline surrounds the entire shape of the subject.