Systems and methods for digital compositing

Abstract

Systems and methods are described for digital compositing. In an example, a video frame from video frames stored in memory can be identified based on video frame latency data. The video frame latency data can specify a number of video frames to be stored in the cache memory space before the video frame is selected. A scene related information frame of the scene related information frames stored in the memory can also be identified based on a timecode of the video frame. Augmented video data that includes one or more composited video frames can be provided based on the video frame and the scene related information frame.

The invention claimed is: 
     
       1. A method comprising:
 receiving scene related information frames for a scene being filmed, each scene related information frame including a timestamp and being provided by a device; 
 receiving video frames representative of the scene, the video frames being provided by a video camera separate from the device; 
 selecting a video frame of the video frames based on video frame latency data; 
 updating the timestamp of each of the scene related information frames based on a timing synchronization factor; 
 selecting a scene related information frame of the scene related information frames with the updated timestamp; and 
 generating an augmented video frame with a digital asset based on the video frame and the scene related information frame. 
 
     
     
       2. The method of  claim 1 , wherein the video frames are provided by the video camera at a different frequency than the scene related information frames. 
     
     
       3. The method of  claim 1 , further comprising extracting a timestamp of the selected video frame, and the scene related information is selected based on the timestamp of the selected video frame. 
     
     
       4. The method of  claim 1 , wherein the video frame is further selected based on a video frame request. 
     
     
       5. The method of  claim 1 , wherein a digital compositor generates the augmented video frame. 
     
     
       6. The method of  claim 1 , wherein a number of the video frames including the video frame is stored in a cache memory space, and the video frame latency data specifying the number of video frames that is to be stored in the cache memory space before the video frame is selected from the cache memory space. 
     
     
       7. The method of  claim 6 , wherein a last stored video frame of the video frames in the cache memory space is selected based on the video frame latency data. 
     
     
       8. The method of  claim 6 , wherein first, second, and third video frames are stored in the cache memory space and the third video frame is selected from the cache memory space. 
     
     
       9. The method of  claim 1 , further comprising:
 creating a virtual environment representative of the scene, the virtual environment including a computer generated image (CGI) element for the digital asset and a virtual camera for the video camera, the scene related information frames specifying a position and/or orientation of the video camera at a moment in time; 
 updating a position of the virtual camera and the CGI element in the virtual environment based on the selected scene related information frame to update the virtual environment; and 
 generating the augmented video frame in response to updating the virtual environment. 
 
     
     
       10. The method of  claim 1 , further comprising determining a network latency of a network, wherein the scene related information frames are provided over the network. 
     
     
       11. The method of  claim 10 , wherein the timing synchronization factor specifies the network latency of the network over which the scene related information frames are provided. 
     
     
       12. The method of  claim 1 , wherein the timing synchronization factor comprises a frame delta value representative of an amount of time between the scene related information frames. 
     
     
       13. The method of  claim 1 , further comprising:
 partitioning a cache memory space into partitions including a first partition and a second partition; 
 storing the video frames in the first partition; and 
 storing the scene related information frames in the second partition. 
 
     
     
       14. The method of  claim 13 , wherein the partitions include a third partition, the method further comprising storing the digital asset in the third partition. 
     
     
       15. The method of  claim 13 , wherein the first partition comprises even and odd video frame locations, and the video frames are stored in one of the even and odd video frame cache locations based on a respective frame number assigned to each of the video frames. 
     
     
       16. The method of  claim 1 , wherein the video frame latency data is provided based on a user input. 
     
     
       17. A system comprising: one or more computing platforms configured to:
 receive scene related information frames for a scene from a first device, each scene related information frame including a timestamp; 
 receive video frames representative of the scene from a second device separate from the first device; 
 select a video frame of the video frames based on video frame latency data; 
 update the timestamp of each the scene related information frames based on a timing synchronization factor; 
 select a scene related information frame of the scene related information frames with the updated timestamp; and 
 generate an augmented video frame with a digital asset based on the video frame and the scene related information frame. 
 
     
     
       18. The system of  claim 17 , wherein the timing synchronization factor specifies a network latency of a network over which the scene related information frames are provided and frame delta value representative of an amount of time between each scene related information frame of the scene related information frames provided by the first device. 
     
     
       19. One or more non-transitory computer-readable media having machine readable instructions executable by a processor, the machine readable instructions being programmed to perform a method comprising:
 receiving scene related information frames for a scene being filmed provided by a first device, each scene related information frame including a timestamp; 
 receiving video frames representative of the scene from a second device separate from the first device; 
 selecting a video frame of the video frames based on video frame latency data; 
 updating the timestamp of each the scene related information frames based on a timing synchronization factor; 
 identifying a scene related information frame of the scene related information frames with the updated timestamp; and 
 providing an augmented video frame with a digital asset based on the video frame and the scene related information frame. 
 
     
     
       20. The one or more non-transitory computer-readable media of  claim 19 , wherein the method further comprises partitioning a cache memory space into partitions including a first partition, a second partition, and a third partition, and wherein:
 the video frames are stored in the first partition, 
 the scene related information frames are stored in the second partition; and 
 the digital asset is stored in the third partition.