Systems, methods, and media for providing interactive refocusing in images

Abstract

Systems, methods, and media for providing interactive refocusing are provided, the systems comprising: a lens; an image sensor; and a processor that: causes the image sensor to capture a plurality of images over a predetermined, period of time, wherein each of the plurality of images represents a scene at a different point in time; changes a depth of field between at least a pair of the plurality of images; concatenates the plurality of images to create a duration focal volume in the order in which the images were captured; computes a space-time in-focus image that represents in-focus portions from each of the plurality of images based on the duration focal volume; and computes a space-time index map that identifies an in-focus image for each location of the scene from among the plurality of images based on die duration focal volume and the space-time in-focus image.

What is claimed: 
     
         1 . A system for providing interactive refocusing, the system comprising:
 a lens;   an image sensor:, and   a hardware processor configured to:
 causes the image sensor to capture a plurality of images over a predetermined period of time, wherein each of the plurality of images represents a scene at a different point in time; 
 changes a depth of field between at least a pair of the plurality of images; 
 concatenates the plurality of images to create a duration focal volume in the order in which the images were captured; 
 computes a space-time in-focus image that represents in-focus portions from each of the plurality of images based on the duration focal volume; and 
 computes a space-time index map that identities an in-focus image for each location of the scene from among the plurality of images based on the duration focal volume and the space-time in-focus image. 
   
     
     
         2 . The system of  claim 1 , wherein changing the depth of field comprises changing a distance between the lens and the image sensor. 
     
     
         3 . The system of  claim 2 , wherein changing the distance between the lens and the image sensor comprises driving an actuator that causes the image sensor to move relative to the lens. 
     
     
         4 . The system of  claim 2 , wherein changing the distance between the lens and the image sensor comprises driving an actuator that causes the lens to move relative to the image sensor. 
     
     
         5 . The system of  claim 1 , wherein changing the depth of field comprises changing a focal length of the lens. 
     
     
         6 . The system of  claim 1 , wherein changing the depth of field comprises moving a second lens in an optical axis direction, wherein the second lens is disposed between the lens and the image sensor in the optical axis direction. 
     
     
         7 . The system of  claim 1 , wherein for each of the plurality of images in the duration focal volume the depth of field of the image is substantially continuous with the depth of field for one or more adjacent images. 
     
     
         8 . The system of  claim 7 , wherein the intersection of the depths of field for the plurality of images is an empty set. 
     
     
         9 . The system of  claim 1 , wherein the plurality of images are captured at a predetermined frame rate. 
     
     
         10 . The system of  claim 9 , wherein the hardware processor is further configured to change a distance between the lens and the image sensor at a constant rate based on at least one of the following parameters: the predetermined frame rate; a pixel size of the image sensor; and an f-number of the lens. 
     
     
         11 . A method for providing interactive refocusing, the method comprising:
 causing, using a hardware processor, a plurality of images to be captured over a predetermined period of time, wherein each of the plurality of images represents a scene at a different point in time;   causing a depth of field at which the plurality of images are captured to be changed between at least a pair of the plurality of images;   create a duration focal volume by concatenating the plurality of images in the order in which the images were captured;   computing a space-time in-focus image that represents in-focus portions from each of the plurality of images based on the duration focal volume; and   computing a space-time index map that identifies an in-focus image for each location of the scene from among the plurality of images based on the duration focal volume and the space-time in-focus image.   
     
     
         12 . The method of  claim 11 , wherein causing the depth of field to be changed comprises causing a distance between a lens used to capture the images and an image sensor used to capture the images to be changed. 
     
     
         13 . The method of  claim 12 , wherein causing the distance to be change comprises driving an actuator that causes the image sensor to move relative to the lens. 
     
     
         14 . The method of  claim 12 , wherein causing the distance to be change comprises driving an actuator that causes the lens to move relative to the image sensor. 
     
     
         15 . The method of  claim 11 , wherein causing the depth of field to be changed comprises causing a foal length of a lens used to capture the images to be changed. 
     
     
         16 . The method of  claim 11 , wherein causing the depth of field to be changed comprises causing a first lens to be moved in an optical axis direction, wherein the first lens is disposed between a second lens used to capture the images and an image sensor used to capture the images in the optical axis direction. 
     
     
         17 . The method of  claim 11 , wherein for each of the plurality of images in the duration focal volume the depth of field of the image is substantially continuous with the depth of field for one or more adjacent images. 
     
     
         18 . The method of  claim 17 , wherein the intersection of the depths of field for the plurality of images is an empty set. 
     
     
         19 . The method of  claim 11 , wherein causing the images to be captured comprises causing the plurality of images to be captured at a predetermined frame rate. 
     
     
         20 . The method of  claim 19 , further comprising causing a distance between a lens used to capture the images and an image sensor used to capture the images to be changed at a constant rate based on at least one of the following parameters: the predetermined frame rate; a pixel size of the image sensor; and an f-number of the lens. 
     
     
         21 . A non-transitory computer readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for providing interactive refocusing, the method comprising:
 causing a plurality of images to be captured over a predetermined period of time wherein each of the plurality of images represents a scene at a different point in time;   causing a depth of field at which the plurality of images are captured to be changed between at least a pair of the plurality of images;   create a duration focal volume by concatenating the plurality of images in the order in which the images were captured;   computing a space-time in-focus image that represents in-focus portions from each of the plurality of images based on the duration focal volume; and   computing a space-time index map that identities an in-focus image for each location of the scene from among the plurality of images based on the duration focal volume and the space-time in-focus image.   
     
     
         22 . The non-transitory computer readable medium of  claim 21 , wherein causing the depth of field to be changed comprises causing a distance between a lens used to capture the images and an image sensor used to capture the images to be changed. 
     
     
         23 . The non-transitory computer readable medium of  claim 22 , wherein causing the distance to be change comprises driving an actuator that causes the image sensor to move relative to the lens. 
     
     
         24 . The non-transitory computer readable medium of  claim 22 , wherein causing the distance to be change comprises driving an actuator that causes the lens to move relative to the image sensor. 
     
     
         25 . The non-transitory computer readable medium of  claim 21 , wherein causing the depth of field to be changed comprises causing a focal length of a lens used to capture the images to be changed. 
     
     
         26 . The non-transitory computer readable medium of  claim 21 , wherein causing the depth of field to be changed comprises causing a first lens to be moved in an optical axis direction, wherein the first lens is disposed between a second lens used to capture the images and an image sensor used to capture the images in the optical axis direction. 
     
     
         27 . The non-transitory computer readable medium of  claim 21 , wherein for each of the plurality of images in the duration focal volume the depth of field of the image is substantially continuous with the depth of field for one or more adjacent images. 
     
     
         28 . The non-transitory computer readable medium of  claim 27 , wherein the intersection of the depths of field for the plurality of images is an empty set. 
     
     
         29 . The non-transitory computer readable medium of  claim 21 , wherein causing the images to be captured comprises causing the plurality of images to be captured at a predetermined frame rate. 
     
     
         30 . The non-transitory computer readable medium of  claim 29 , the method further comprising causing a distance between a lens used to capture the images and an image sensor used to capture the images to be changed at a constant rate based on at least one of the following parameters: the predetermined frame rate; a pixel size of the image sensor; and an f-number of the lens. 
     
     
         31 . A system for providing interactive refocusing, the system comprising:
 a lens;   an image sensor;   means for causing the image sensor to capture a plurality of images over a predetermined period of time, wherein each of the plurality of images represents a scene at a different point in time;   means for changing a depth of field between at least a pair of the plurality of images;   means for concatenating the plurality of images to create a duration focal volume in the order in which the images were captured;   means for computing a space-time in-focus image that represents in-focus portions from each of the plurality of images based on the duration focal volume; and   means for computing a space-time index map that identifies an in-focus image for each location of the scene from among the plurality of images based on the duration focal volume and the space-time in-focus image.   
     
     
         32 . The system of  claim 31 , wherein the means for changing the depth of field comprises means for changing a distance between the lens and the image sensor. 
     
     
         33 . The system of  claim 32 , wherein the means for changing the distance between the lens and the image sensor comprises means for driving an actuator that causes the image sensor to move relative to the lens. 
     
     
         34 . The system of  claim 32 , wherein the means for changing the distance between the lens and the image sensor comprises means for driving an actuator that causes the lens to move relative to the image sensor. 
     
     
         35 . The system of  claim 31 , wherein the means for changing the depth of field comprises means for changing a focal length of the lens. 
     
     
         36 . The system of  claim 31 , wherein the means for changing the depth of field comprises means for moving a second lens in an optical axis direction, wherein the second lens is disposed between the lens and the image sensor in the optical axis direction. 
     
     
         37 . The system of  claim 31 , wherein for each of the plurality of images in the duration focal volume the depth of field of the image is substantially continuous with the depth of field for one or more adjacent images. 
     
     
         38 . The system of  claim 37 , wherein the intersection of the depths of field for the plurality of images is an empty set. 
     
     
         39 . The system of  claim 31 , further comprising means for causing the plurality of images to be captured at a predetermined frame rate. 
     
     
         40 . The system of  claim 39 , further comprising means for changing a distance between the lens and the image sensor at a constant rate based on at least one of the following parameters: the predetermined frame rate; a pixel size of the image sensor; and an f-number of the lens. 
     
     
         41 . A system for providing interactive refocusing, the system comprising:
 a hardware processor configured to:
 causes an image of a scene to be displayed to a user, wherein the image is representative of the scene represented in a duration focal volume that comprises a plurality of images captured of the scene at different points in time, and wherein the plurality of images are captured at a plurality of different depths of field; 
 receives a selection of a point in the image by the user; 
 causes a selected image from among the plurality of images to be displayed to the user, wherein the selected image is an image where an object in the scene at the point selected by the user is in focus. 
   
     
     
         42 . The system of  claim 41 , wherein the hardware processor is further configured to cause one or more intermediate images captured between the capturing of the image of the scene and capturing of the selected image to be displayed to the user prior to displaying the selected image. 
     
     
         43 . A method for providing interactive refocusing, the method comprising:
 causing an image of a scene to be displayed to a user, wherein the image is representative of the scene represented in a duration focal volume that comprises a plurality of images captured of the scene at different points in time, and wherein the plurality of images are captured at a plurality of different depths of field;   receiving a selection of a point in the image by the user;   causing a selected image from among the plurality of images to be displayed to the user, wherein the selected image is an image where an object in the scene at the point selected by the user is in focus.   
     
     
         44 . The method of  claim 43 , further comprising causing one or more intermediate images captured between the capturing of the image of the scene and capturing of the selected image to be displayed to the user prior to displaying the selected image. 
     
     
         45 . A non-transitory computer readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for providing interactive refocusing, the method comprising:
 causing an image of a scene to be displayed to a user, wherein the image is representative of the scene represented in a duration focal volume that comprises a plurality of images captured of the scene at different points in time, and wherein the plurality of images are captured at a plurality of different depths of field;   receiving a selection of a point in the image by the user;   causing a selected image from among the plurality of images to be displayed to the user, wherein the selected image is an image where an object in the scene at the point selected by the user is in focus.   
     
     
         46 . The non-transitory computer readable medium of  claim 45 , further comprising, causing one or more intermediate images captured between the capturing of the image of the scene and capturing of the selected image to be displayed to the user prior to displaying the selected image. 
     
     
         47 . A method for providing an interactive refocusing image, the method comprising:
 receiving, using a hardware processor, a plurality of images of a scene captured at different points in time, wherein each of the plurality of images represents a different depth of field of the scene;   concatenating the plurality of images to create a duration focal volume in the order in which the images were captured;   computing a space-time in-focus image that represents in-focus portions from each of the plurality of images based on the duration focal volume; and   computing a space-time index map that identifies an in-focus image for each location of the scene front among the plurality of images based on the duration focal volume and the space-time in-focus image.   
     
     
         48 . The method of  claim 47 , wherein computing the space-time index map comprises:
 comparing the duration focal volume and the space-time in-focus image at various scales;   storing a depth map corresponding to each scale level, wherein each depth map is created based on the comparing at the corresponding scale level; and   merging the depth maps based on the space-time in-focus image.   
     
     
         49 . The method of  claim 48 , wherein merging the depth maps comprises:
 computing a reliable depth map based on the depth maps corresponding to each reliable scale level;   determining segments in the in-focus image;   creating a plane for each segment based on depths from the reliable depth map for pixels in the segment;   determining whether the plane is reliable based on an error between the plane and depths from the reliable depth map; and   upon determining that the plane is reliable, filling the depth map based on the plane.   
     
     
         50 . A method for providing an interactive refocusing image, the method comprising:
 receiving, using a hardware processor, a plurality of images of a scene captured at different points in time, wherein each of the plurality of images represents a different depth of field of the scene;   concatenating the plurality of images to create a duration focal volume in the order in which the images were captured; and   for each point (x,y,P 0)  in the duration focal volume:   (a) setting a point P in the duration focal volume equal to P 0;      (b) selecting a cone centered at P;   (c) finding a point Q in the cone at which a measure of focus is highest;   (d) determining whether Q equals P;   (e) repeating (b) through (d) until it is determined, that Q equals P; and   (f) setting a depth for point (x,y, P 0 ) equal to Q.   
     
     
         51 . A method for providing an interactive refocusing image, the method comprising:
 receiving, using a hardware processor, a plurality of images of a scene captured at different points in time, wherein each of the plurality of images represents a different depth of field of the scene;   concatenating the plurality of images to create a duration focal volume in the order in which the images were captured;   determining an optical flow between each pair of successive images;   determining a pixel trajectory through the duration focal volume for each pixel location in the scene;   determining an in-focus point for each pixel by determining a point along the pixel trajectory where a measure of focus for the pixel is highest; and   computing a space-time index map that identifies an in-focus image for each pixel of the scene from among the plurality of images based the in-focus point for each pixel.