Video generation using convict hulls

Abstract

Video of a scene is generated and presented to a user. A stream of mesh models of the scene is generated from one or more streams of sensor data that represent the scene. Each of the mesh models is sliced using a series of planes that are parallel to each other, where each of the planes in the series defines one or more contours each of which defines a specific region on the plane where the mesh model intersects the plane. A texture map is generated for each of the mesh models which defines texture data corresponding to each of the contours that is defined by the series of planes. Images of the scene are rendered from scene proxies that include a stream of mathematical equations describing the contours, and a stream of the texture maps. The images are displayed.

Wherefore, what is claimed is: 
     
         1 . A computer-implemented process for generating a video of a scene, comprising:
 using a computing device to perform the following process actions:   receiving one or more streams of sensor data that represent the scene; and   generating scene proxies from said streams of sensor data, said generation comprising the actions of:
 generating a stream of mesh models of the scene from said streams of sensor data, and 
 for each of the mesh models,
 slicing the mesh model using a series of planes that are parallel to each other, each of the planes in the series defining one or more contours each of which defines a specific region on the plane where the mesh model intersects the plane, and 
 generating a texture map for the mesh model which defines texture data corresponding to each of the contours that is defined by the series of planes. 
 
   
     
     
         2 . The process of  claim 1 , wherein the texture map for the mesh model comprises a series of scanlines each of which corresponds to a different one of the planes in the series of planes, each of the scanlines comprises a series of texels, and the process action of generating a texture map for the mesh model which defines texture data corresponding to each of the contours that is defined by the series of planes comprises the actions of:
 for each of the planes in the series of planes,
 analyzing each of the contours that is defined by the plane in a prescribed order across the plane to identify a series of point locations along the contour, said analysis being performed starting from a prescribed zero position on the contour, said zero position being the same for each of the contours that is defined by the plane, 
 for each of the contours that is defined by the plane, using the series of point locations to determine a mathematical equation describing the contour, 
 assigning the texels in the scanline corresponding to the plane to the contours that are defined by the plane, said texel assignment being performed in the prescribed order across the plane, and 
 entering information specifying said texel assignment into the texture map for the mesh model; and 
   using the one or more streams of sensor data that represent the scene to compute texture data for each of the texels that is in the texture map for the mesh model; and   entering said computed texture data into the texture map for the mesh model.   
     
     
         3 . The process of  claim 2 , wherein the process action of using the one or more streams of sensor data that represent the scene to compute texture data for each of the texels that is in the texture map for the mesh model comprises an action of, for each of said texels, using a projective texture mapping method to sample each of said streams of sensor data and combine texture information from each of said samples to generate texture data for the texel. 
     
     
         4 . The process of  claim 2 , wherein the process action of assigning the texels in the scanline corresponding to the plane to the contours that are defined by the plane comprises the actions of:
 for each of the contours that is defined by the plane,
 calculating the length of the contour, 
 calculating the normalized length of the contour by dividing the length of the contour by the sum of the lengths of all of the contours that are defined by the plane, 
 calculating the number of texels in said scanline that are to be assigned to the contour by multiplying the normalized length of the contour and the total number of texels that is in said scanline, and 
 assigning said calculated number of texels to the contour. 
   
     
     
         5 . The process of  claim 1 , wherein the texture data comprises one or more of: color data; or specular highlight data; or transparency data; or reflection data; or shadowing data. 
     
     
         6 . The process of  claim 1 , wherein each of the mesh models comprises a collection of vertices, a prescribed spacing is used between successive planes in the series of planes, and said spacing is selected such that the series of planes intersects a maximum number of vertices in each of the mesh models. 
     
     
         7 . The process of  claim 1 , wherein either,
 the one or more streams of sensor data comprise a single stream of sensor data which represents the scene from a single geometric perspective, and the video being generated is a single viewpoint video, or   the one or more streams of sensor data comprise a plurality of streams of sensor data each of which represents the scene from a different geometric perspective, and the video being generated is a free viewpoint video.   
     
     
         8 . The process of  claim 1 , further comprising an action of storing the scene proxies, said storing comprising the actions of:
 for each of the mesh models,
 storing a mathematical equation describing each of the contours that is defined by the series of planes, 
 storing data specifying which contours on neighboring planes in the series of planes correspond to each other, and 
 storing the texture map for the mesh model. 
   
     
     
         9 . The process of  claim 8 , wherein the mathematical equation describing a given contour specifies either a polygon approximation of the contour, or a non-uniform rational basis spline curve approximation of the contour. 
     
     
         10 . The process of  claim 8  wherein,
 whenever the spatial orientation of the series of planes is not pre-determined, the process action of storing the scene proxies further comprises an action of storing data specifying said spatial orientation, and 
 whenever the geometry of the series of planes is not pre-determined, the process action of storing the scene proxies further comprises an action of storing data specifying said geometry, said data comprising one or more of:
 data specifying the number of planes in the series of planes; or 
 data specifying a prescribed spacing that is used between successive planes in the series of planes; or 
 data specifying the shape and dimensions of each of the planes in the series of planes. 
 
 
     
     
         11 . The process of  claim 8  wherein,
 whenever the prescribed order across the plane is not pre-determined, the process action of storing the scene proxies further comprises an action of storing data specifying said order, 
 whenever the number of texels in each of the scanlines is not pre-determined, the process action of storing the scene proxies further comprises an action of storing data specifying said number, and 
 whenever the prescribed zero position on the contour is not pre-determined, the process action of storing the scene proxies further comprises an action of storing data specifying said zero position. 
 
     
     
         12 . The process of  claim 1 , further comprising an action of distributing the scene proxies to an end user who either is, or will be, viewing the video on another computing device which is connected to a data communication network, said distribution comprising the actions of:
 for each of the mesh models,
 transmitting a mathematical equation describing each of the contours that is defined by the series of planes over the network to said other computing device, 
 transmitting data specifying which contours on neighboring planes in the series of planes correspond to each other over the network to said other computing device, and 
 transmitting the texture map for the mesh model over the network to said other computing device. 
   
     
     
         13 . The process of  claim 12 , wherein whenever the spatial orientation of the series of planes is not pre-determined, the process action of distributing the scene proxies to an end user who either is, or will be, viewing the video on another computing device which is connected to a data communication network further comprises an action of transmitting data specifying said spatial orientation over the network to said other computing device. 
     
     
         14 . The process of  claim 12 , wherein whenever the geometry of the series of planes is not pre-determined, the process action of distributing the scene proxies to an end user who either is, or will be, viewing the video on another computing device which is connected to a data communication network further comprises an action of transmitting data specifying said geometry over the network to said other computing device, said data comprising one or more of:
 data specifying the number of planes in the series of planes; or   data specifying a prescribed spacing that is used between successive planes in the series of planes; or   data specifying the shape and dimensions of each of the planes in the series of planes.   
     
     
         15 . The process of  claim 12 , wherein,
 whenever the prescribed order across the plane is not pre-determined, the process action of distributing the scene proxies to an end user who either is, or will be, viewing the video on another computing device which is connected to a data communication network further comprises an action of transmitting data specifying said order over the network to said other computing device,   whenever the number of texels in each of the scanlines is not pre-determined, the process action of distributing the scene proxies to an end user who either is, or will be, viewing the video on another computing device which is connected to a data communication network further comprises an action of transmitting data specifying said number over the network to said other computing device, and   whenever the prescribed zero position on the contour is not pre-determined, the process action of distributing the scene proxies to an end user who either is, or will be, viewing the video on another computing device which is connected to a data communication network further comprises an action of transmitting data specifying said zero position over the network to said other computing device.   
     
     
         16 . The process of  claim 1 , wherein the series of planes comprises either a horizontal spatial orientation or a vertical spatial orientation. 
     
     
         17 . A computer-implemented process for presenting a video of a scene to a user, comprising:
 using a computing device to perform the following process actions:   receiving scene proxies, said scene proxies comprising:
 a stream of mathematical equations describing contours that are defined by a series of planes that are parallel to each other, and 
 a stream of texture maps defining texture data corresponding to each of the contours that is defined by the series of planes; 
   rendering images of the scene from the scene proxies, said rendering comprising the actions of:
 constructing the series of planes using data specifying the spatial orientation and geometry of the series of planes, 
 constructing the contours that are defined by the series of planes using the stream of mathematical equations, 
 constructing a series of point locations along each of said contours, said construction being performed in a prescribed order across each of the planes in the series of planes, said construction also being performed starting from a prescribed zero position on each of said contours, 
 tessellating the point locations that are defined by the series of planes, said tessellation generating a stream of polygonal models, each polygonal model comprising a collection of polygonal faces that are formed by neighboring point locations on corresponding contours on neighboring planes in the series of planes, 
 sampling the stream of texture maps to identify the texture data that corresponds to each of the polygonal faces in the stream of polygonal models, and 
 using said identified texture data to add texture to each of the polygonal faces in the stream of polygonal models; and 
   displaying the images of the scene.   
     
     
         18 . The process of  claim 17 , wherein each of the texture maps in the stream of texture maps comprises a series of scanlines each of which corresponds to a different one of the planes in the series of planes, each of the scanlines comprises a series of texels that are assigned to each of the contours that is defined by the plane corresponding to the scanline, and the process action of sampling the stream of texture maps to identify the texture data that corresponds to each of the polygonal faces in the stream of polygonal models comprises the actions of:
 for each of the scanlines in each of the texture maps, adapting the number of texels in the scanline that are assigned to each one of the contours that is defined by the plane corresponding to the scanline to be the average of the number of texels in the scanline that are assigned to said one of the contours and the number of texels in the next scanline in the series of scanlines that are assigned to a contour that corresponds to said one of the contours, said adaption resulting in a modified version of each of the texture maps; and   sampling the modified version of each of the texture maps to identify the texture data that corresponds to each of the polygonal faces in the stream of polygonal models.   
     
     
         19 . The process of  claim 17 , wherein the video being presented comprises one of:
 asynchronous single viewpoint video; or   asynchronous free viewpoint video; or   unidirectional live single viewpoint video; or   unidirectional live free viewpoint video; or   bidirectional live single viewpoint video; or   bidirectional live free viewpoint video.   
     
     
         20 . A computer-implemented process for generating a video of a scene, comprising:
 using a computing device to perform the following process actions:   receiving one or more streams of sensor data that represent the scene;   generating scene proxies from said streams of sensor data, said scene proxies generation comprising the actions of:
 generating a stream of mesh models of the scene from said streams of sensor data, and 
 for each of the mesh models,
 slicing the mesh model using a series of planes that are parallel to each other, each of the planes in the series defining one or more contours each of which defines a specific region on the plane where the mesh model intersects the plane, and 
 generating a texture map for the mesh model which defines texture data corresponding to each of the contours that is defined by the series of planes, said texture map comprising a series of scanlines each of which corresponds to a different one of the planes in the series of planes, each of the scanlines comprising a series of texels, said texture map generation comprising the actions of,
 for each of the planes in the series of planes, 
  analyzing each of the contours that is defined by the plane in a prescribed order across the plane to identify a series of point locations along the contour, 
  for each of the contours that is defined by the plane, using the series of point locations to determine a mathematical equation describing the contour, 
  assigning the texels in the scanline corresponding to the plane to the contours that are defined by the plane, said texel assignment being performed in the prescribed order across the plane, and 
  entering information specifying said texel assignment into the texture map for the mesh model, 
 using the one or more streams of sensor data that represent the scene to compute texture data for each of the texels that is in the texture map for the mesh model, and 
 entering said computed texture data into the texture map for the mesh model; and 
 
 
   distributing the scene proxies to an end user who either is, or will be, viewing the video on another computing device which is connected to a data communication network, said distribution comprising the actions of:
 for each of the mesh models,
 transmitting a mathematical equation describing each of the contours that is defined by the series of planes over the network to said other computing device, 
 transmitting data specifying which contours on neighboring planes in the series of planes correspond to each other over the network to said other computing device, and 
 transmitting the texture map for the mesh model over the network to said other computing device.