Light Field Imaging Device and Method for Depth Acquisition and Three-Dimensional Imaging

Abstract

A light field imaging device and method are provided. The device can include a diffraction grating assembly receiving a wavefront from a scene and including one or more diffraction gratings, each having a grating period along a grating axis and diffracting the wavefront to generate a diffracted wavefront. The device can also include a pixel array disposed under the diffraction grating assembly and detecting the diffracted wavefront in a near-field diffraction regime to provide light field image data about the scene. The pixel array has a pixel pitch along the grating axis that is smaller than the grating period. The device can further include a color filter array disposed over the pixel array to spatio-chromatically sample the diffracted wavefront prior to detection by the pixel array. The device and method can be implemented in backside-illuminated sensor architectures. Diffraction grating assemblies for use in the device and method are also disclosed.

1 . A light field imaging device for capturing light field image data about a scene, the light field imaging device comprising:
 a diffraction grating assembly configured to receive an optical wavefront originating from the scene, the diffraction grating assembly comprising a diffraction grating having a grating axis and a refractive index modulation pattern having a grating period along the grating axis, the diffraction grating diffracting the optical wavefront to generate a diffracted wavefront; and   a pixel array comprising a plurality of light-sensitive pixels disposed under the diffraction grating assembly and detecting the diffracted wavefront as the light field image data, the pixel array having a pixel pitch along the grating axis that is smaller than the grating period.   
     
     
         2 . The light field imaging device of  claim 1 , further comprising a color filter array disposed over the pixel array and comprising a plurality of color filters arranged in a mosaic color pattern, the color filter array spatially and spectrally filtering the diffracted wavefront according to the mosaic color pattern prior to detection of the diffracted wavefront by the plurality of light-sensitive pixels. 
     
     
         3 . The light field imaging device of  claim 2 , wherein each color filter is optically coupled to a corresponding one of the light-sensitive pixels. 
     
     
         4 . The light field imaging device of  claim 2 , wherein each color filter is optically coupled to at least two corresponding ones of the plurality of light-sensitive pixels. 
     
     
         5 . The light field imaging device of  claim 2 , wherein each color filter is one of a red pass filter, a green pass filter and a blue pass filter. 
     
     
         6 . The light field imaging device of  claim 2 , wherein the mosaic color pattern is a Bayer pattern. 
     
     
         7 . The light field imaging device of  claim 1 , wherein the diffraction grating is configured to diffract the optical wavefront in a waveband ranging from 400 nanometers to 1550 nanometers. 
     
     
         8 . The light field imaging device of  claim 1 , wherein the grating period ranges from 1 micrometer to 20 micrometers. 
     
     
         9 . The light field imaging device of  claim 1 , wherein the diffraction grating includes between two and ten repetitions of the grating period. 
     
     
         10 . The light field imaging device of  claim 1 , wherein the diffraction grating is a phase grating. 
     
     
         11 . The light field imaging device of  claim 10 , wherein the diffraction grating is a binary phase grating and the refractive index modulation pattern comprises a series of ridges periodically spaced-apart at the grating period, interleaved with a series of grooves periodically spaced-apart at the grating period. 
     
     
         12 . The light field imaging device of  claim 11 , wherein the diffraction grating has a duty cycle of about 50%. 
     
     
         13 . The light field imaging device of  claim 12 , wherein each light-sensitive pixel is positioned under and in alignment with either a corresponding one of the ridges or a corresponding one of the grooves. 
     
     
         14 . The light field imaging device of  claim 12 , wherein each light-sensitive pixel is positioned under and in alignment with a transition between a corresponding one of the ridges and a corresponding adjacent one of the grooves. 
     
     
         15 . The light field imaging device of  claim 11 , wherein the diffraction grating has a duty cycle different from 50%. 
     
     
         16 . The light field imaging device of  claim 11 , wherein the series of ridges has a step height with respect to the series of grooves, the step height ranging from 0.2 micrometer to 1 micrometer. 
     
     
         17 . The light field imaging device of  claim 1 , wherein a separation distance between the refractive index modulation pattern of the diffraction grating and a light-receiving surface of the pixel array ranges from 0.5 micrometer to 20 micrometers. 
     
     
         18 . The light field imaging device of  claim 1 , wherein a separation distance between the refractive index modulation pattern of the diffraction grating and a light-receiving surface of the pixel array is less than about ten times a center wavelength of the optical wavefront. 
     
     
         19 . The light field imaging device of  claim 1 , wherein a ratio of the grating period to the pixel pitch along the grating axis is substantially equal to two. 
     
     
         20 . The light field imaging device of  claim 1 , wherein the plurality of light-sensitive pixels is arranged in a rectangular pixel grid defined by two orthogonal pixel axes, and wherein the grating axis is either parallel to one of the two orthogonal pixel axes or oblique to both the two orthogonal pixel axes. 
     
     
         21 . The light field imaging device of  claim 1 , wherein the pixel pitch ranges from 1 micrometer to 10 micrometers. 
     
     
         22 . The light field imaging device of  claim 1 , further comprising dispersive optics disposed in a light path of the optical wavefront between the scene and the diffraction grating assembly, the dispersive optics being configured to receive and spectrally disperse the optical wavefront. 
     
     
         23 . The light field imaging device of  claim 1 , further comprising a microlens array disposed over the pixel array and comprising a plurality of microlenses, each microlens being optically coupled to a corresponding one of the light-sensitive pixels. 
     
     
         24 . The light field imaging device of  claim 1 , further comprising pixel array circuitry disposed either under the pixel array, in a backside illumination configuration, or between the diffraction grating assembly and the pixel array, in a frontside illumination configuration. 
     
     
         25 . The light field imaging device of  claim 1 , wherein the diffraction grating is one of a plurality of diffraction gratings of the diffraction grating assembly, the plurality of diffraction gratings being arranged in a two-dimensional grating array disposed over the pixel array. 
     
     
         26 . The light field imaging device of  claim 25 , wherein the plurality of diffraction gratings comprises multiple sets of diffraction gratings, the grating axes of the diffraction gratings of different ones of the sets having different orientations. 
     
     
         27 . The light field imaging device of  claim 26 , wherein the multiple sets of diffraction gratings comprise a first set of diffraction gratings and a second set of diffraction gratings, the grating axes of the diffraction gratings of the first set extending substantially perpendicularly to the grating axes of the diffraction gratings of the second set. 
     
     
         28 . The light field imaging device of  claim 24 , wherein each diffraction grating comprises a grating substrate including a top surface having the refractive index modulation pattern formed thereon, the grating substrate comprising a spectral filter material or region configured to spectrally filter the diffracted wavefront prior to detection of the diffracted wavefront by the plurality of light-sensitive pixels, the plurality of diffraction gratings thus forming a color filter array. 
     
     
         29 . The light field imaging device of  claim 28 , wherein the grating substrate of each diffraction grating acts as one of a red pass filter, a green pass filter and a blue pass filter. 
     
     
         30 . The light field imaging device of  claim 28 , wherein the color filter array is arranged in a Bayer pattern. 
     
     
         31 . A backside-illuminated light field imaging device for capturing light field image data about a scene, the backside-illuminated light field imaging device comprising:
 a substrate having a front surface and a back surface;   a diffraction grating assembly disposed over the back surface of the substrate and configured to receive an optical wavefront originating from the scene, the diffraction grating assembly comprising a diffraction grating having a grating axis and a refractive index modulation pattern having a grating period along the grating axis, the diffraction grating diffracting the optical wavefront to generate a diffracted wavefront;   a pixel array formed in the substrate and comprising a plurality of light-sensitive pixels configured to receive through the back surface and detect as the light field image data the diffracted wavefront, the pixel array having a pixel pitch along the grating axis that is smaller than the grating period; and   pixel array circuitry disposed under the front surface and coupled to the pixel array.   
     
     
         32 . The backside-illuminated light field imaging device of  claim 31 , further comprising a color filter array disposed over the back surface and comprising a plurality of color filters arranged in a mosaic color pattern, the color filter array spatially and spectrally filtering the diffracted wavefront according to the mosaic color pattern prior to detection of the diffracted wavefront by the plurality of light-sensitive pixels. 
     
     
         33 . The backside-illuminated light field imaging device of  claim 32 , wherein the mosaic color pattern is a Bayer pattern. 
     
     
         34 . The backside-illuminated light field imaging device of  claim 31 , wherein the diffraction grating is a binary phase grating and the refractive index modulation pattern comprises a series of ridges periodically spaced-apart at the grating period, interleaved with a series of grooves periodically spaced-apart at the grating period. 
     
     
         35 . The backside-illuminated light field imaging device of  claim 34 , wherein the diffraction grating has a duty cycle of about 50% and each light-sensitive pixel is positioned under and in alignment with either a corresponding one of the ridges or a corresponding one of the grooves. 
     
     
         36 . The backside-illuminated light field imaging device of  claim 34 , wherein the diffraction grating has a duty cycle of about 50% and each light-sensitive pixel is positioned under and in alignment with a transition between a corresponding one of the ridges and a corresponding adjacent one of the grooves. 
     
     
         37 . The backside-illuminated light field imaging device of  claim 31 , wherein a separation distance between the refractive index modulation pattern of the diffraction grating and a light-receiving surface of the pixel array ranges from 0.5 micrometer to 5 micrometers. 
     
     
         38 . The backside-illuminated light field imaging device of  claim 31 , wherein a ratio of the grating period to the pixel pitch along the grating axis is substantially equal to two. 
     
     
         39 . The backside-illuminated light field imaging device of  claim 31 , wherein the plurality of light-sensitive pixels is arranged in a rectangular pixel grid defined by two orthogonal pixel axes, and wherein the grating axis is either parallel to one of the two orthogonal pixel axes or oblique to both the two orthogonal pixel axes. 
     
     
         40 . The backside-illuminated light field imaging device of  claim 31 , wherein the pixel pitch ranges from 1 micrometer to 5 micrometers. 
     
     
         41 . The backside-illuminated light field imaging device of  claim 31 , further comprising dispersive optics disposed in a light path of the optical wavefront between the scene and the diffraction grating assembly, the dispersive optics being configured to receive and spectrally disperse the optical wavefront. 
     
     
         42 . The backside-illuminated light field imaging device of  claim 31 , further comprising a microlens array disposed over the pixel array and comprising a plurality of microlenses, each microlens being optically coupled to a corresponding one of the light-sensitive pixels. 
     
     
         43 . The backside-illuminated light field imaging device of  claim 31 , wherein the diffraction grating is one of a plurality of diffraction gratings of the diffraction grating assembly, the plurality of diffraction gratings being arranged in a two-dimensional grating array disposed over the pixel array. 
     
     
         44 . The backside-illuminated light field imaging device of  claim 43 , wherein the plurality of diffraction gratings comprises multiple sets of diffraction gratings, the grating axes of the diffraction gratings of different ones of the sets having different orientations. 
     
     
         45 . The backside-illuminated light field imaging device of  claim 44 , wherein the multiple sets of diffraction gratings comprise a first set of diffraction gratings and a second set of diffraction gratings, the grating axes of the diffraction gratings of the first set extending substantially perpendicularly to the grating axes of the diffraction gratings of the second set. 
     
     
         46 . The backside-illuminated light field imaging device of  claim 31 , further comprising:
 a color filter array disposed over the back surface and comprising a plurality of color filters, each of which optically coupled to a corresponding one of the plurality of light-sensitive pixels, the color filter array spatially and spectrally filtering the diffracted wavefront prior to detection of the diffracted wavefront by the plurality of light-sensitive pixels; and   a microlens array disposed over the color filter array and comprising a plurality of microlenses, each microlens being optically coupled to a corresponding one of the plurality of the color filters,   wherein the diffraction grating further comprises a grating substrate including a top surface having the refractive index modulation pattern formed thereon and a bottom surface disposed over the microlens array.   
     
     
         47 . A light field imaging device comprising:
 a diffraction grating assembly comprising a diffraction grating having a grating axis and a refractive index modulation pattern having a grating period along the grating axis; and   a pixel array comprising a plurality of light-sensitive pixels disposed under the diffraction grating, the pixel array having a pixel pitch along the grating axis that is smaller than the grating period.   
     
     
         48 . The light field imaging device of  claim 47 , further comprising a color filter array disposed over the pixel array and comprising a plurality of color filters arranged in a mosaic color pattern, the color filter array spatially and spectrally filtering the diffracted wavefront according to the mosaic color pattern prior to detection of the diffracted wavefront by the plurality of light-sensitive pixels. 
     
     
         49 . The light field imaging device of  claim 47 , wherein the grating period ranges from 1 micrometer to 20 micrometers. 
     
     
         50 . The light field imaging device of  claim 47 , wherein the diffraction grating is a binary phase grating and the refractive index modulation pattern comprises a series of ridges periodically spaced-apart at the grating period, interleaved with a series of grooves periodically spaced-apart at the grating period. 
     
     
         51 . The light field imaging device of  claim 50 , wherein the diffraction grating has a duty cycle of about 50% and each light-sensitive pixel is positioned under and in alignment with either a corresponding one of the ridges or a corresponding one of the grooves. 
     
     
         52 . The light field imaging device of  claim 50 , wherein the diffraction grating has a duty cycle of about 50% and each light-sensitive pixel is positioned under and in alignment with a transition between a corresponding one of the ridges and a corresponding adjacent one of the grooves. 
     
     
         53 . The light field imaging device of  claim 47 , wherein a ratio of the grating period to the pixel pitch along the grating axis is substantially equal to two. 
     
     
         54 . The light field imaging device of  claim 47 , wherein the plurality of light-sensitive pixels is arranged in a rectangular pixel grid defined by two orthogonal pixel axes, and wherein the grating axis is either parallel to one of the two orthogonal pixel axes or oblique to both the two orthogonal pixel axes. 
     
     
         55 . The light field imaging device of  claim 47 , further comprising dispersive optics disposed in a light path of the optical wavefront between the scene and the diffraction grating assembly, the dispersive optics being configured to receive and spectrally disperse the optical wavefront. 
     
     
         56 . The light field imaging device of  claim 47 , wherein the diffraction grating is one of a plurality of diffraction gratings of the diffraction grating assembly, the plurality of diffraction gratings being arranged in a two-dimensional grating array disposed over the pixel array. 
     
     
         57 . The light field imaging device of  claim 56 , wherein the plurality of diffraction gratings comprises multiple sets of diffraction gratings, the grating axes of the diffraction gratings of different ones of the sets having different orientations. 
     
     
         58 . The light field imaging device of  claim 57 , wherein the multiple sets of diffraction gratings comprise a first set of diffraction gratings and a second set of diffraction gratings, the grating axes of the diffraction gratings of the first set extending substantially perpendicularly to the grating axes of the diffraction gratings of the second set. 
     
     
         59 . A diffraction grating assembly for use with an image sensor comprising a pixel array having a plurality of light-sensitive pixels to capture light field image data about a scene, the diffraction grating assembly comprising a diffraction grating having a grating axis and a refractive index modulation pattern having a grating period along the grating axis, the grating period being larger than a pixel pitch of the pixel array along the grating axis, the diffraction grating being configured to receive and diffract an optical wavefront originating from the scene to generate a diffracted wavefront for detection by the light-sensitive pixels as the light field image data, the diffraction grating assembly being configured to be disposed over the pixel array. 
     
     
         60 . The diffraction grating assembly of  claim 59 , wherein the diffraction grating assembly is configured to be disposed over a color filter array of the image sensor, the color filter array being disposed over pixel array and configured to spatially and spectrally filter the diffracted wavefront prior to detection of the diffracted wavefront by the plurality of light-sensitive pixels. 
     
     
         61 . The diffraction grating assembly of  claim 59 , wherein the diffraction grating is configured to diffract the optical wavefront in a waveband ranging from 400 nanometers to 1550 nanometers. 
     
     
         62 . The diffraction grating assembly of  claim 59 , wherein the grating period ranges from 1 micrometer to 20 micrometers. 
     
     
         63 . The diffraction grating assembly of  claim 59 , wherein the diffraction grating is a binary phase grating and the refractive index modulation pattern comprises a series of ridges periodically spaced-apart at the grating period, interleaved with a series of grooves periodically spaced-apart at the grating period. 
     
     
         64 . The diffraction grating assembly of  claim 59 , wherein a ratio of the grating period to the pixel pitch along the grating axis is substantially equal to two. 
     
     
         65 . The diffraction grating assembly of  claim 59 , wherein the diffraction grating is one of a plurality of diffraction gratings of the diffraction grating assembly, the plurality of diffraction gratings being arranged in a two-dimensional grating array disposed over the pixel array. 
     
     
         66 . The diffraction grating assembly of  claim 65 , wherein the plurality of diffraction gratings comprises multiple sets of diffraction gratings, the grating axes of the diffraction gratings of different ones of the sets having different orientations. 
     
     
         67 . The diffraction grating assembly of  claim 66 , wherein the multiple sets of diffraction gratings comprise a first set of diffraction gratings and a second set of diffraction gratings, the grating axes of the diffraction gratings of the first set extending substantially perpendicularly to the grating axes of the diffraction gratings of the second set. 
     
     
         68 . The diffraction grating assembly of  claim 59 , wherein each diffraction gratings includes between two and ten repetitions of the grating period. 
     
     
         69 . A method of capturing light field image data about a scene, the method comprising:
 diffracting an optical wavefront originating from the scene with a diffraction grating having a grating period along a grating axis to generate a diffracted wavefront; and   detecting the diffracted wavefront as the light field image data with a pixel array comprising a plurality of light-sensitive pixels disposed under the diffraction grating, the pixel array having a pixel pitch along the grating axis that is smaller than the grating period.   
     
     
         70 . The method of  claim 69 , further comprising spatio-spectrally filtering the diffracted wavefront with a color filter array prior to detecting the diffracted wavefront with the plurality of light-sensitive pixels. 
     
     
         71 . The method of  claim 69 , wherein diffracting the optical wavefront originating from the scene comprises diffracting the optical wavefront in a waveband ranging from 400 nanometers to 1550 nanometers. 
     
     
         72 . The method of  claim 69 , further comprising selecting the grating period in a range between 1 micrometer to 20 micrometers. 
     
     
         73 . The method of  claim 69 , further comprising providing the diffraction grating as a binary phase grating comprising a series of ridges periodically spaced-apart at the grating period, interleaved with a series of grooves periodically spaced-apart at the grating period. 
     
     
         74 . The method of  claim 73 , further comprising providing the diffraction grating with a duty cycle of about 50% and positioning each light-sensitive pixel under and in alignment with either a corresponding one of the ridges or a corresponding one of the grooves, or under and in alignment with a transition between a corresponding one of the ridges and a corresponding adjacent one of the grooves. 
     
     
         75 . The method of  claim 73 , further comprising setting a step height of the ridges relative to the grooves to control an optical path difference between adjacent ones of the ridges and grooves. 
     
     
         76 . The method of  claim 69 , further comprising setting a separation distance between the refractive index modulation pattern of the diffraction grating and a light-receiving surface of the pixel array to less than about ten times a center wavelength of the optical wavefront. 
     
     
         77 . The method of  claim 69 , further comprising setting a ratio of the grating period to the pixel pitch along the grating axis to be substantially equal to two. 
     
     
         78 . The method of  claim 69 , further comprising providing the plurality of light-sensitive pixels in a rectangular pixel grid defined by two orthogonal pixel axes, and orienting the grating axis either parallel to one of the two orthogonal pixel axes or oblique to both the two orthogonal pixel axes. 
     
     
         79 . The method of  claim 69 , further comprising spectrally dispersing the optical wavefront prior to diffracting the optical wavefront. 
     
     
         80 . A method of providing three-dimensional imaging capabilities to an image sensor viewing a scene and comprising a pixel array having a plurality of light-sensitive pixels, the method comprising:
 disposing a diffraction grating assembly in front of the image sensor, the diffraction grating assembly comprising a diffraction grating having a grating axis and a grating period along the grating axis, the grating period being larger than a pixel pitch of the pixel array along the grating axis;   receiving and diffracting an optical wavefront originating from the scene with the diffraction grating to generate a diffracted wavefront; and   detecting the diffracted wavefront with the light-sensitive pixels.   
     
     
         81 . The method of  claim 80 , further comprising spatio-spectrally filtering the diffracted wavefront with a color filter array prior to detecting the diffracted wavefront by the plurality of light-sensitive pixels. 
     
     
         82 . The method of  claim 80 , further comprising selecting the grating period in a range between 1 micrometer to 20 micrometers. 
     
     
         83 . The method of  claim 80 , further comprising providing the diffraction grating as a binary phase grating comprising a series of ridges periodically spaced-apart at the grating period, interleaved with a series of grooves periodically spaced-apart at the grating period. 
     
     
         84 . The method of  claim 83 , further comprising providing the diffraction grating with a duty cycle of about 50% and positioning the diffraction grating assembly over the pixel array such that either each ridge and each groove extends over and in alignment with a corresponding one of the light-sensitive pixels, or each transition between the interleaved ridges and grooves extends over and in alignment with a corresponding one of the light-sensitive pixels. 
     
     
         85 . The method of  claim 83 , further comprising setting a step height of the ridges relative to the grooves to control an optical path difference between adjacent ones of the ridges and grooves. 
     
     
         86 . The method of  claim 80 , wherein disposing the diffraction grating assembly in front of the image sensor comprises positioning the diffraction grating assembly at a separation distance from the pixel array selected such that an optical path length of the diffracted wavefront prior to being detected with the light-sensitive pixels is less than about ten times a center wavelength of the optical wavefront. 
     
     
         87 . The method of  claim 80 , further comprising setting the grating period equal to substantially twice the pixel pitch along the grating axis. 
     
     
         88 . The method of  claim 80 , wherein disposing the diffraction grating assembly in front of the image sensor comprises orienting the grating axis either parallel to one of two orthogonal pixel axes of the pixel array or oblique to both the two orthogonal pixel axes. 
     
     
         89 . The method of  claim 80 , further comprising spectrally dispersing the optical wavefront prior to diffracting the optical wavefront.