View frustum culling for free viewpoint video (fvv)

Abstract

The view frustum culling technique described herein allows Free Viewpoint Video (FVV) or other 3D spatial video rendering at a client by sending only the 3D geometry and texture (e.g., RGB) data necessary for a specific viewpoint or view frustum from a server to the rendering client. The synthetic viewpoint is then rendered by the client by using the received geometry and texture data for the specific viewpoint or view frustum. In some embodiments of the view frustum culling technique, the client has both some texture data and 3D geometric data stored locally if there is sufficient local processing power. Additionally, in some embodiments, additional spatial and temporal data can be sent to the client to support changes in the view frustum by providing additional geometry and texture data that will likely be immediately used if the viewpoint is changed either spatially or temporally.

What is claimed is: 
     
         1 . A computer-implemented process for receiving spatial three dimensional video, comprising:
 using a client computing device for:
 receiving only texture data and geometric data for a given view frustum of a spatial three dimensional video from a server at a client; 
 rendering the given viewpoint of the spatial three dimensional video at the client using the downloaded texture and geometric data for the given view frustum. 
   
     
     
         2 . The computer-implemented process of  claim 1  wherein the client specifies the given view frustum to the server before the texture data and geometric data are downloaded to the client. 
     
     
         3 . The computer-implemented process of  claim 1  wherein the client receives texture data and geometric data computed by the server based on a viewpoint received from the client. 
     
     
         4 . The computer-implemented process of  claim 1 , further comprising:
 checking if texture data or geometric data has been previously downloaded to the client; and   not downloading the texture data or the geometric data which has previously downloaded to the client again.   
     
     
         5 . The computer-implemented process of  claim 1  wherein additional texture data and corresponding geometric data at the edges of the view frustum is received at the client. 
     
     
         6 . The computer-implemented process of  claim 1  wherein the client's viewpoint is predicted based on the client's rate of viewpoint change. 
     
     
         7 . The computer-implemented process of  claim 6  wherein the view frustum is expanded based on the client's predicted viewpoint. 
     
     
         8 . The computer-implemented process of  claim 6  wherein a lower level of detail of geometric data is received at the client for viewpoints that the client has a lower probability of reaching. 
     
     
         9 . The computer-implemented process of  claim 1  wherein a lower level of detail of texture data and geometric data is sent for objects in the distance of the client's view frustum. 
     
     
         10 . The computer-implemented process of  claim 1  wherein the geometric data is stored as a spatial representation of all viewpoints possible. 
     
     
         11 . The computer-implemented process of  claim 1  wherein the spatial representation of all viewpoints possible is defined by three dimensional cells. 
     
     
         12 . The computer-implemented process of  claim 11  wherein the server stores the cells and wherein the client requests specific cells corresponding to a desired view point to be rendered. 
     
     
         13 . The computer-implemented process of  claim 11  wherein the server computes the cells to send to the client based on a viewpoint the client wishes to render. 
     
     
         14 . The computer-implemented process of  claim 11  wherein the three dimensional cells are in a compressed format. 
     
     
         15 . A computer-implemented process for receiving free viewpoint video, comprising:
 using a client computing device for:
 installing a free viewpoint video player on a local client; 
 selecting a free viewpoint video stored on a server; 
 receiving a message from the server that tells the client to instantiate the free viewpoint video player with controls appropriate to the selected free viewpoint video type; 
 instantiating the free viewpoint video player with controls appropriate to the selected free viewpoint video type; 
 requesting a desired viewpoint of the selected free viewpoint video from the server; 
 receiving only the necessary geometric and texture data to render the desired viewpoint of the selected viewpoint video; and 
 combining the received geometric and texture data to render the desired viewpoint of the free viewpoint video 
   
     
     
         16 . The computer-implemented process of  claim 15  further comprising:
 the client checking for user viewpoint navigation input; and 
 if there is any user viewpoint navigation input, the client sending the navigation input to the server. 
 
     
     
         17 . The computer-implemented process of  claim 16  wherein the server uses the client's navigation input to determine which 3D geometry and texture data to next send to the client. 
     
     
         18 . A system for providing free viewpoint video, comprising:
 a general purpose computing device;   a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,   download only texture data and geometric data relevant to a given viewpoint of a free viewpoint video at a client;   render the given viewpoint of the free viewpoint video at the client using only the downloaded texture and geometric data for the given viewpoint.   
     
     
         19 . The system of  claim 18  wherein the downloaded texture data and the downloaded geometric data is downloaded from more than one server in a computing cloud. 
     
     
         20 . The system of  claim 17  wherein the downloaded texture data and geometric data is slightly greater than required to render the given viewpoint.