Blended rendering of focused plenoptic camera data

Abstract

Methods, apparatus, and computer-readable storage media for rendering focused plenoptic camera data. A rendering with blending technique is described that blends values from positions in multiple microimages and assigns the blended value to a given point in the output image. A rendering technique that combines depth-based rendering and rendering with blending is also described. Depth-based rendering estimates depth at each microimage and then applies that depth to determine a position in the input flat from which to read a value to be assigned to a given point in the output image. The techniques may be implemented according to parallel processing technology that renders multiple points of the output image in parallel. In at least some embodiments, the parallel processing technology is graphical processing unit (GPU) technology.

What is claimed is: 
     
       1. A method, comprising:
 obtaining a flat comprising a plurality of microimages of separate portions of an image of a scene, wherein each of the plurality of microimages is in a separate region of the flat that is a two-dimensional (2D) representation of a four-dimensional (4D) light-field that captures both spatial and angular information of the scene; 
 rendering an output image of the scene from the plurality of microimages, said rendering for each point of the output image comprising: 
 determining a target microimage in the flat for a point; 
 calculating a position in the target microimage from which a value is to be sampled; 
 for each microimage in a neighborhood including the target microimage, reading the value from the calculated position in the respective microimage and adding the value to a value accumulator, said adding the value comprising weighting the value read from the respective microimage according to a position of the respective microimage in the neighborhood prior to said adding the value to the value accumulator; and 
 normalizing the value in the value accumulator and assigning the normalized value to the point in the output image. 
 
     
     
       2. The method as recited in  claim 1 , wherein said rendering the output image of the scene from the plurality of microimages comprises performing said rendering according to a parallel processing technique that renders multiple points of the output image in parallel. 
     
     
       3. The method as recited in  claim 2 , wherein the parallel processing technique is implemented according to graphical processing unit (GPU) technology. 
     
     
       4. The method as recited in  claim 1 , wherein said normalizing the value in the value accumulator comprises normalizing the value in the value accumulator according to a sum of the weights of the microimages in the neighborhood. 
     
     
       5. The method as recited in  claim 1 , wherein said calculating the position in the target microimage from which the value is to be sampled comprises calculating the position within an M×M region within the target microimage. 
     
     
       6. The method as recited in  claim 1 , wherein for each said microimage in the neighborhood including the target microimage, the values from the calculated positions in the respective microimages are averaged together across the plurality of microimages. 
     
     
       7. The method as recited in  claim 1 , wherein a focused plenoptic camera implements said obtaining a plenoptic image as the flat comprising the plurality of the microimages through an array of microlenses. 
     
     
       8. A non-transitory computer-readable storage memory storing program instructions that are executable and, responsive to execution of the program instructions, a computer device implements operations comprising to:
 obtaining a flat comprising a plurality of microimages of separate portions of an image of a scene, wherein each of the plurality of microimages is in a separate region of the flat that is a two-dimensional (2D) representation of a four-dimensional (4D) light-field that captures both spatial and angular information of the scene; 
 rendering an output image of the scene from the plurality of microimages, said rendering for each point of the output image comprising: 
 determining a target microimage in the flat for a point; 
 calculating a position in the target microimage from which a value is to be sampled; 
 for each microimage in a neighborhood including the target microimage, reading the value from the calculated position in the respective microimage and adding the value to a value accumulator, said adding the value comprising weighting the value read from the respective microimage according to a position of the respective microimage in the neighborhood prior to said adding the value to the value accumulator; and 
 normalizing the value in the value accumulator and assigning the normalized value to the point in the output image. 
 
     
     
       9. The non-transitory computer-readable storage memory as recited in  claim 8 , wherein the computer device implements said rendering the output image of the scene from the plurality of microimages according to a parallel processing technique that renders multiple points of the output image in parallel. 
     
     
       10. The non-transitory computer-readable storage memory as recited in  claim 9 , wherein the parallel processing technique is implemented according to graphical processing unit (GPU) technology. 
     
     
       11. The non-transitory computer-readable storage memory as recited in  claim 8 , wherein the computer device implements said normalizing the value in the value accumulator according to a sum of the weights of the microimages in the neighborhood. 
     
     
       12. The non-transitory computer-readable storage memory as recited in  claim 8 , wherein the computer device implements said calculating the position in the target microimage from which the value is to be sampled within an M×M region within the target microimage. 
     
     
       13. The computer-readable storage memory as recited in  claim 8 , wherein for each said microimage in the neighborhood including the target microimage, the computer device implements averaging the values from the calculated positions in the respective microimages together across the plurality of microimages. 
     
     
       14. The computer-readable storage memory as recited in  claim 8 , wherein the computer device is a focused plenoptic camera that implements said obtaining a plenoptic image as the flat comprising the plurality of the microimages through an array of microlenses. 
     
     
       15. A system, comprising
 at least one processor; and 
 a memory comprising program instructions that are executable by the at least one processor to: 
 obtain a flat comprising a plurality of microimages of separate portions of an image of a scene, wherein each of the plurality of microimages is in a separate region of the flat that is a two-dimensional (2D) representation of a four-dimensional (4D) light-field that captures both spatial and angular information of the scene; 
 render an output image of the scene from the plurality of microimages, and for each point of the output image: 
 determine a target microimage in the flat for a point; 
 calculate a position in the target microimage from which a value is to be sampled; 
 for each microimage in a neighborhood including the target microimage, read the value from the calculated position in the respective microimage and add the value to a value accumulator, the value being added comprising the value read from the respective microimage weighted according to a position of the respective microimage in the neighborhood prior to the value being added to the value accumulator; and 
 normalize the value in the value accumulator and assign the normalized value to the point in the output image. 
 
     
     
       16. The system as recited in  claim 15 , wherein the at least one processor includes at least one graphical processing unit (GPU), and wherein the program instructions are executable by the at least one GPU to render the output image of the scene from the plurality of microimages according to a parallel processing technique that renders multiple points of the output image in parallel. 
     
     
       17. The system as recited in  claim 15 , wherein the program instructions are executable by the at least one processor to normalize the value in the value accumulator according to a sum of the weights of the microimages in the neighborhood. 
     
     
       18. The system as recited in  claim 15 , wherein the program instructions are executable by the at least one processor to calculate the position in the target microimage from which the value is to be sampled within an M×M region within the target microimage. 
     
     
       19. The system as recited in  claim 15 , wherein for each said microimage in the neighborhood including the target microimage, the program instructions are executable by the at least one processor to average the values from the calculated positions in the respective microimages together across the plurality of microimages. 
     
     
       20. The system as recited in  claim 15 , wherein a focused plenoptic camera comprises the at least one processor and the memory, and wherein the program instructions are executable to said obtain a plenoptic image as the flat comprising the plurality of the microimages through an array of microlenses.