Cinematic space-time view synthesis for enhanced viewing experiences in computing environments

Abstract

A mechanism is described for facilitating cinematic space-time view synthesis in computing environments according to one embodiment. A method of embodiments, as described herein, includes capturing, by one or more cameras, multiple images at multiple positions or multiple points in times, where the multiple images represent multiple views of an object or a scene, where the one or more cameras are coupled to one or more processors of a computing device. The method further includes synthesizing, by a neural network, the multiple images into a single image including a middle image of the multiple images and representing an intermediary view of the multiple views.

What is claimed is: 
     
       1. An apparatus comprising:
 at least one memory; 
 instructions in the apparatus; and 
 processor circuitry to execute the instructions to:
 access input images of an object or scene captured by one or more cameras, different ones of the input images associated with at least one of different times of capture or different perspectives of the object or scene to define a space-time volume of images; 
 synthesize pairs of the input images along a cinematic path to generate intermediate images corresponding to respective ones of the pairs, the cinematic path associated with a subset of the input images within the space-time volume of images, synthesis of the pairs not based on calibration data of the one or more cameras and not based on position information of the one or more cameras; and 
 generate a series of frames for a video based on the intermediate images. 
 
 
     
     
       2. The apparatus of  claim 1 , wherein the processor circuitry is to synthesize the pairs without different ones of the one or more cameras being temporally synchronized. 
     
     
       3. An apparatus comprising:
 at least one memory; 
 instructions in the apparatus; and 
 processor circuitry to execute the instructions to:
 access input images of an object or scene captured by one or more cameras, different ones of the input images associated with at least one of different times of capture or different perspectives of the object or scene to define a space-time volume of images, the input images including first and second input images; 
 synthesize the first and second input images to generate a first one of intermediate images by:
 generating displacement data based on a change of position of corresponding pixels in the first and second input images; 
 generating first and second warped images based on the displacement data, the first warped image corresponding to a warped version of the first input image, the second warped image corresponding to a warped version of the second input image; and 
 combining the first and second warped images to produce the first intermediate image, the first and second input images are a first one of pairs of the input images synthesized along a cinematic path to generate the intermediate images corresponding to respective ones of the pairs, the cinematic path associated with a subset of the input images within the space-time volume of images; and 
 
 generate a series of frames for a video based on the intermediate images. 
 
 
     
     
       4. The apparatus of  claim 3 , wherein the first intermediate image corresponds to at least one of an intermediate time of capture between the different times of capture of the first and second input images or an intermediate perspective between the different perspectives of the first and second input images. 
     
     
       5. The apparatus of  claim 3 , wherein the displacement data includes a first displacement map and a second displacement map, the first displacement map indicative of a change of position of first pixels in the first input image relative to corresponding ones of second pixels in the second input image, the second displacement map indicative of a change of position of third pixels in the second input image relative to corresponding ones of fourth pixels in the first input image. 
     
     
       6. The apparatus of  claim 5 , wherein the processor circuitry is to generate the first warped image based on the first displacement map and generate the second warped image based on the second displacement map. 
     
     
       7. At least one storage device comprising instructions that, when executed, cause processor circuitry to at least:
 access input images of an object or scene captured by one or more cameras, different ones of the input images associated with at least one of different times of capture or different perspectives of the object or scene to define a space-time volume of images; 
 synthesize successive pairs of the input images along a cinematic path to generate intermediate images corresponding to respective ones of the successive pairs, the cinematic path passing through a subset of input images within the space-time volume of images, synthesis of the successive pairs not dependent on different ones of the one or more cameras being temporally synchronized when the input images were captured; and 
 generate a series of frames for a video based on the subset of input images and the intermediate images. 
 
     
     
       8. The at least one storage device of  claim 7 , wherein the instructions cause the processor circuitry to synthesize the successive pairs without using calibration data of the one or more cameras and without using position information of the one or more cameras. 
     
     
       9. At least one storage device comprising instructions that, when executed, cause processor circuitry to at least:
 access input images of an object or scene captured by one or more cameras, different ones of the input images associated with at least one of different times of capture or different perspectives of the object or scene to define a space-time volume of images, the input images including first and second input images; 
 synthesize the first and second input images to generate a first one of intermediate images by:
 generating displacement data based on a change of position of corresponding pixels in the first and second input images; 
 generating first and second warped images based on the displacement data, the first warped image corresponding to a warped version of the first input image, the second warped image corresponding to a warped version of the second input image; and 
 combining the first and second warped images to produce the first intermediate image, the first and second input images are a first pair of successive pairs of the input images synthesized along a cinematic path to generate intermediate images corresponding to respective ones of the successive pairs, the cinematic path passing through a subset of input images within the space-time volume of images; and 
 
 generate a series of frames for a video based on the subset of input images and the intermediate images. 
 
     
     
       10. The at least one storage device of  claim 9 , wherein the first intermediate image corresponds to at least one of an intermediate time of capture between the different times of capture of the first and second input images or an intermediate perspective between the different perspectives of the first and second input images. 
     
     
       11. The at least one storage device of  claim 9 , wherein the displacement data includes a first displacement map and a second displacement map, the first displacement map indicative of a change of position of first pixels in the first input image relative to corresponding ones of second pixels in the second input image, the second displacement map indicative of a change of position of third pixels in the second input image relative to corresponding ones of fourth pixels in the first input image. 
     
     
       12. The at least one storage device of  claim 11 , wherein the instructions cause the processor circuitry to generate the first warped image based on the first displacement map and generate the second warped image based on the second displacement map. 
     
     
       13. An apparatus comprising:
 means for storing instructions; and 
 means for executing the instructions, the executing means to:
 access input images of an object or scene captured by one or more cameras, different ones of the input images associated with at least one of different times of capture or different perspectives of the object or scene, the different ones of the input images to collectively define a space-time volume of images; 
 synthesize pairs of a subset of the input images identified by a cinematic path extending through the space-time volume of images, the synthesis of the pairs to generate intermediate images for respective ones of the pairs, synthesis of the pairs not based on calibration data of the one or more cameras and not based on position information of the one or more cameras; and 
 generating a series of frames for a video that includes the intermediate images. 
 
 
     
     
       14. The apparatus of  claim 13 , wherein the executing means is to synthesize the pairs without different ones of the one or more cameras being temporally synchronized. 
     
     
       15. An apparatus comprising:
 means for storing instructions; and 
 means for executing the instructions, the executing means to:
 access input images of an object or scene captured by one or more cameras, different ones of the input images associated with at least one of different times of capture or different perspectives of the object or scene, the different ones of the input images to collectively define a space-time volume of images, the input images including first and second input images; 
 synthesize the first and second input images to generate a first one of intermediate images by:
 generating displacement data based on a change of position of corresponding pixels in the first and second input images; 
 generating first and second warped images based on the displacement data, the first warped image corresponding to a warped version of the first input image, the second warped image corresponding to a warped version of the second input image; and 
 combining the first and second warped images to produce the first intermediate image, the first and second input images corresponding to a first one of pairs of a subset of the input images identified by a cinematic path extending through the space-time volume of images, the synthesis of the pairs to generate intermediate images for respective ones of the pairs; and 
 
 
 generating a series of frames for a video that includes the intermediate images. 
 
     
     
       16. The apparatus of  claim 15 , wherein the first intermediate image corresponds to at least one of an intermediate time of capture between the different times of capture of the first and second input images or an intermediate perspective between the different perspectives of the first and second input images. 
     
     
       17. The apparatus of  claim 15 , wherein the displacement data includes a first displacement map and a second displacement map, the first displacement map indicative of a change of position of first pixels in the first input image relative to corresponding ones of second pixels in the second input image, the second displacement map indicative of a change of position of third pixels in the second input image relative to corresponding ones of fourth pixels in the first input image. 
     
     
       18. The apparatus of  claim 17 , wherein the executing means is to generate the first warped image based on the first displacement map and generate the second warped image based on the second displacement map.