Capturing light-field volume image and video data using tiled light-field cameras

Abstract

A capture system may capture light-field data representative of an environment for use in virtual reality, augmented reality, and the like. The system may have a plurality of light-field cameras arranged to capture a light-field volume within the environment, and a processor. The processor may use the light-field volume to generate a first virtual view depicting the environment from a first virtual viewpoint. The light-field cameras may be arranged in a tiled array to define a capture surface with a ring-shaped, spherical, or other arrangement. The processor may map the pixels captured by the image sensors to light rays received in the light-field volume, and store data descriptive of the light rays in a coordinate system representative of the light-field volume.

What is claimed is: 
     
       1. A system comprising:
 a plurality of light-field cameras arranged to create a substantially gap-free surface in a tiled array so as to capture a substantially fully sampled light-field volume within an environment, wherein each of the light-field cameras comprises an image sensor that captures pixels representative of the environment; and 
 a processor configured to:
 map the pixels to light rays received in the light-field volume; 
 store data descriptive of the light rays in a coordinate system representative of the light-field volume, wherein the coordinate system comprises a polar coordinate system and the light rays are recorded in terms of a first angle of intersection with an inner sphere, a second angle of intersection with the inner sphere, a first angle of intersection with an outer sphere, and a second angle of intersection with the outer sphere; and 
 
 use the substantially fully sampled light-field volume to generate a first virtual view depicting the environment from a first virtual viewpoint. 
 
     
     
       2. The system of  claim 1 , wherein the light-field cameras are arranged such that outward-facing elements of the light-field cameras define a physical capture surface larger than the light-field volume;
 wherein the physical capture surface encircles the light-field volume. 
 
     
     
       3. The system of  claim 2 , wherein the processor is further configured to interpolate between portions of the light-field volume captured by the light-field cameras to generate additional data to fill holes in the light-field volume resulting from gaps between the light-field cameras. 
     
     
       4. The system of  claim 1 , wherein at least a subset of the light-field cameras are arranged in a ring array to define a generally cylindrical capture surface that captures the light-field volume. 
     
     
       5. The system of  claim 4 , wherein the light-field cameras are arranged in multiple layers in the ring array to define the generally cylindrical capture surface. 
     
     
       6. The system of  claim 4 , wherein at least one of light-field cameras is positioned on an axis of the generally cylindrical capture surface and is oriented substantially perpendicularly to the light-field cameras of the subset. 
     
     
       7. The system of  claim 1 , wherein the light-field cameras are arranged in a spherical array to define a generally spherical capture surface that captures the light-field volume. 
     
     
       8. The system of  claim 1 , wherein each of the light-field cameras comprises an image sensor and a microlens array that projects light from the light-field volume onto the image sensor. 
     
     
       9. The system of  claim 8 , wherein each of the light-field cameras further comprises a tapered fiber optic bundle comprising a leading end facing toward the environment and a trailing end positioned proximate the image sensor. 
     
     
       10. The system of  claim 1 , wherein:
 each of the light-field cameras comprises an objective lens through which light from the light-field volume enters the light-field camera; and 
 the objective lenses of the light-field cameras are compactly arranged to minimize gaps between the objective lenses. 
 
     
     
       11. The system of  claim 1 , wherein:
 each of the light-field cameras comprises an objective lens through which light from the light-field volume enters the light-field camera; and 
 each of the objective lenses comprises a double Gauss lens. 
 
     
     
       12. The system of  claim 1 , further comprising a motor that rotates the light-field cameras relative to the environment. 
     
     
       13. The system of  claim 1 , wherein each of the light-field cameras comprises an objective lens through which the light-field volume enters the light-field camera and an image sensor that captures the light-field volume;
 and wherein the system further comprises a motor that rotates the image sensors of the light-field cameras relative to the objective lenses of the light-field cameras. 
 
     
     
       14. The system of  claim 1 , wherein each of the light-field cameras comprises an array light-field camera. 
     
     
       15. The system of  claim 1 , wherein the processor is further configured to generate the first virtual view by:
 positioning a virtual lens and a virtual image sensor on the coordinate system; and 
 modeling passage of the light rays through the virtual lens to the virtual image sensor. 
 
     
     
       16. The system of  claim 1 , wherein:
 the processor is further configured to use the light-field volume to generate a second virtual view depicting the environment from a second virtual viewpoint; and 
 the second virtual viewpoint is spaced apart from the first virtual viewpoint such that the first virtual view and the second virtual view cooperate to simulate viewing of the environment by a user. 
 
     
     
       17. A system comprising:
 a plurality of center view cameras arranged in at least a first circular pattern around an axis, wherein each of the center view cameras has a field-of-view generally perpendicular to the axis; 
 a plurality of upper view cameras arranged in at least a second circular pattern around the axis, wherein each of the upper view cameras has a field-of-view oriented generally upward; and 
 a plurality of lower view cameras arranged in at least a third circular pattern around the axis, wherein each of the lower view cameras has a field-of-view oriented generally downward; and 
 wherein the first circular pattern is interleaved with at least one of the second circular pattern and the third circular pattern to define a first ring circling the axis, in which each of the center view cameras in the first ring is positioned between two elements in the first ring, wherein the two elements are selected from the group consisting of the upper view cameras and the lower view cameras. 
 
     
     
       18. The system of  claim 17 , further comprising a processor configured to use image data captured by the center view cameras, the upper view cameras, and the lower view cameras from within an environment to generate a first virtual view depicting the environment from a first virtual viewpoint. 
     
     
       19. The system of  claim 17 , wherein the first circular pattern is interleaved with the second circular pattern and the third circular pattern to define the first ring such that-each of the center view cameras in the first ring is positioned between one of the upper view cameras in the first ring and one of the lower view cameras in the first ring. 
     
     
       20. The system of  claim 17 , wherein:
 the first circular pattern is interleaved with the second circular pattern to define the first ring such that-each of the center view cameras in the first ring is positioned between two of the upper view cameras in the first ring; and 
 the center view cameras are further arranged in a fourth circular pattern interleaved with the third circular pattern to define a second ring circling the axis, such that-each of the center view cameras in the second ring is positioned between two of the lower view cameras in the second ring. 
 
     
     
       21. The system of  claim 17 , wherein:
 each of the center view cameras comprises a center view camera field-of-view; 
 each of the upper view cameras comprises an upper view camera field-of-view; 
 each of the lower view cameras comprises a lower view camera field-of-view; and 
 the upper view camera field-of-view and the lower view camera field-of-view are larger than the center view camera field-of-view. 
 
     
     
       22. A method of generating one or more virtual viewpoints of an environment, the method comprising:
 at a plurality of light-field cameras arranged to create a substantially gap-free surface in a tiled array, capturing a substantially fully sampled light-field volume within the environment, wherein each of the light-field cameras comprises an image sensor and capturing the light-field volume comprises, with each of the light-field cameras, using the image sensor to capture pixels representative of the environment; 
 at a processor, mapping the pixels to light rays received in the light-field volume; 
 at the processor, storing data descriptive of the light rays in a coordinate system representative of the light-field volume, wherein the coordinate system comprises a polar coordinate system and wherein storing data descriptive of the light rays in the coordinate system comprises recording the light rays in terms of a first angle of intersection with an inner sphere, a second angle of intersection with the inner sphere, a first angle of intersection with an outer sphere, and a second angle of intersection with the outer sphere; and 
 at the processor, using the substantially fully sampled light-field volume to generate a first virtual view depicting the environment from a first virtual viewpoint. 
 
     
     
       23. The method of  claim 22 , further comprising, at the processor, interpolating between portions of the light-field volume captured by the light-field cameras to generate additional data to fill holes in the light-field volume resulting from gaps between the light-field cameras. 
     
     
       24. The method of  claim 22 , wherein:
 each of the light-field cameras comprises an image sensor and a microlens array; and 
 capturing the light-field volume comprises, with each of the light-field cameras, projecting light with the microlens array from the light-field volume onto the image sensor. 
 
     
     
       25. The method of  claim 24 , wherein:
 each of the light-field cameras further comprises a tapered fiber optic bundle comprising a leading end facing toward the environment and a trailing end positioned proximate the image sensor; and 
 capturing the light-field volume further comprises, with each of the light-field cameras, guiding the light to the image sensor by causing the light to pass from the leading end to the trailing end. 
 
     
     
       26. The method of  claim 22 , wherein:
 each of the light-field cameras comprises an objective lens; 
 the objective lenses of the light-field cameras are compactly arranged to minimize gaps between the objective lenses; and 
 capturing the light-field volume comprises, with each of the light-field cameras, receiving light from the light-field volume through the objective lens. 
 
     
     
       27. The method of  claim 22 , wherein using the light-field volume to generate the first virtual view comprises:
 positioning a virtual lens and a virtual image sensor on the coordinate system; and 
 modeling passage of the light rays through the virtual lens to the virtual image sensor. 
 
     
     
       28. The method of  claim 22 , further comprising, at the processor, using the light-field volume to generate a second virtual view depicting the environment from a second virtual viewpoint;
 wherein the second virtual viewpoint is spaced apart from the first virtual viewpoint such that the first virtual view and the second virtual view cooperate to simulate viewing of the environment by a user. 
 
     
     
       29. A method of generating one or more virtual viewpoints of an environment, the method comprising:
 at a plurality of cameras arranged in a tiled array, capturing image data from within the environment; and 
 at a processor, using the image data to generate a first virtual view depicting the environment from a first virtual viewpoint; 
 wherein the plurality of cameras comprises:
 a plurality of center view cameras arranged in at least a first circular pattern around an axis, wherein each of the center view cameras has a field-of-view generally perpendicular to the axis; 
 a plurality of upper view cameras arranged in at least a second circular pattern around the axis, wherein each of the upper view cameras has a field-of-view oriented generally upward; and 
 a plurality of lower view cameras arranged in at least a third circular pattern around the axis, wherein each of the lower view cameras has a field-of-view oriented generally downward; and 
 wherein the first circular pattern is interleaved with at least one of the second circular pattern and the third circular pattern to define a first ring circling the axis, in which each of the center view cameras in the first ring is positioned between two elements in the first ring, wherein the two elements are selected from the group consisting of the upper view cameras and the lower view cameras. 
 
 
     
     
       30. The method of  claim 29 , wherein the first circular pattern is interleaved with the second circular pattern and the third circular pattern to define the first ring such that-each of the center view cameras in the first ring is positioned between one of the upper view cameras in the first ring and one of the lower view cameras in the first ring. 
     
     
       31. The method of  claim 29 , wherein:
 the first circular pattern is interleaved with the second circular pattern to define the first ring such that-each of the center view cameras in the first ring is positioned between two of the upper view cameras in the first ring; and 
 the center view cameras are further arranged in a fourth circular pattern interleaved with the third circular pattern to define a second ring circling the axis, such that-each of the center view cameras in the second ring is positioned between two of the lower view cameras in the second ring. 
 
     
     
       32. The method of  claim 29 , wherein:
 each of the center view cameras comprises a center view camera field-of-view; 
 each of the upper view cameras comprises an upper view camera field-of-view; 
 each of the lower view cameras comprises a lower view camera field-of-view; and 
 the upper view camera field-of-view and the lower view camera field-of-view are larger than the center view camera field-of-view.