Real time production display of composited images with use of mutliple-source image data

Abstract

Compositing is provided in which visual elements from different sources, including live action objects and computer graphic (CG) merged in a constant feed. Representative output images are produced during a live action shoot. The compositing system uses supplementary data, such as depth data of the live action objects for integration with CG items and light marker detection data for device calibration and performance capture. Varying capture times (e.g., exposure times) and processing times are tracked to align with corresponding incoming images and data.

We claim: 
     
         1 . A computer-implemented method for using a representative output images during recording of a live action scene for a visual production, the method comprising:
 during recording of the live action scene, producing representative output images of the live action scene composited with computer graphics (“CG”) elements, wherein producing the representative output images comprises:
 determining that marker data from marker images generated by a performance capture system and received during first time blocks, correspond with auxiliary images with depth data received during second time blocks; and 
 compositing the CG elements based on the depth data, with the auxiliary image; and 
   providing the representative output images to a display device for viewing on location during the recording of the live action scene.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 receiving updated production information altered in response to display of the representative output images on the display device, wherein the updated production information is stored in a central storage unit.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein the updated production information includes compositional changes in recording the live action scene and the method further comprises:
 altering at least one CG element of the CG elements in response to the updated production information.   
     
     
         4 . The computer-implemented method of  claim 3 , wherein the compositional changes include at least one of a change in a lighting source in the live action scene or a change in parameters of one or more capture devices recording the live action scene. 
     
     
         5 . The computer-implemented method of  claim 3 , wherein the compositional changes are extracted image characteristics from the auxiliary images, including at least one of the depth data, focal length, zoom level, or lighting effects. 
     
     
         6 . The computer-implemented method of  claim 2 , wherein the updated production information is stored as current production information and the central storage unit further includes prior outdated production information. 
     
     
         7 . The computer-implemented method of  claim 1 , wherein the display device further displays live action images of the live action scene presently being captured by a capture device. 
     
     
         8 . A visual content composition system for correlating data and images from a live action scene in real time, the system comprising:
 one or more processors; and   logic encoded in one or more tangible media for execution by the one or more processors and when executed is operable to perform operations comprising:
 during recording of the live action scene, producing representative output images of the live action scene composited with computer graphics (“CG”) elements, wherein producing the representative output images comprises:
 determining that marker data from marker images generated by a performance capture system and received during first time blocks, correspond with auxiliary images with depth data received during second time blocks; and 
 compositing the CG elements based on the depth data, with the auxiliary image; and 
 
 providing the representative output images to a display device for viewing on location during the recording of the live action scene. 
   
     
     
         9 . The visual content composition system of  claim 8 , wherein the operations further comprise:
 receiving updated production information altered in response to display of the representative output images on the display device, wherein the updated production information is stored in a central storage unit.   
     
     
         10 . The visual content composition system of  claim 9 , wherein the updated production information includes compositional changes in recording the live action scene and the operations further comprise:
 altering at least one CG element of the CG elements in response to the updated production information.   
     
     
         11 . The visual content composition system of  claim 10 , wherein the compositional changes include at least one of a change in a lighting source in the live action scene or a change in parameters of one or more capture devices recording the live action scene. 
     
     
         12 . The visual content composition system of  claim 10 , wherein the compositional changes are extracted image characteristics from the auxiliary images, including at least one of the depth data, focal length, zoom level, or lighting effects. 
     
     
         13 . The visual content composition system of  claim 9 , wherein the updated production information is stored as current production information and the central storage unit further includes prior outdated production information. 
     
     
         14 . The visual content composition system of  claim 12 , wherein the display device further displays live action images of the live action scene presently being captured by a capture device. 
     
     
         15 . A non-transitory computer-readable storage medium carrying program instructions thereon to correlate data and images from a live action scene in real time, the instructions when executed by one or more processors cause the one or more processors to perform operations comprising:
 during recording of the live action scene, producing representative output images of the live action scene composited with computer graphics (“CG”) elements, wherein producing the representative output images comprises:
 determining that marker data from marker images generated by a performance capture system and received during first time blocks, correspond with auxiliary images with depth data received during second time blocks; and 
 compositing the CG elements based on the depth data, with the auxiliary image; and 
   providing the representative output images to a display device for viewing on location during the recording of the live action scene.   
     
     
         16 . The non-transitory computer-readable storage medium of  claim 15 , wherein the operations further comprise:
 receiving updated production information altered in response to display of the representative output images on the display device, wherein the updated production information is stored in a central storage unit.   
     
     
         17 . The non-transitory computer-readable storage medium of  claim 16 , wherein the updated production information includes compositional changes in recording the live action scene and the operations further comprise:
 altering at least one CG element of the CG elements in response to the updated production information.   
     
     
         18 . The non-transitory computer-readable storage medium of  claim 17 , wherein the compositional changes include at least one of a change in a lighting source in the live action scene or a change in parameters of one or more capture devices recording the live action scene. 
     
     
         19 . The non-transitory computer-readable storage medium of  claim 17 , wherein the compositional changes are extracted image characteristics from the auxiliary images, including at least one of the depth data, focal length, zoom level, or lighting effects. 
     
     
         20 . The non-transitory computer-readable storage medium of  claim 16 , wherein the updated production information is stored as current production information and the central storage unit further includes prior outdated production information.