System and method for motion warping using multi-exposure frames

Abstract

A method includes obtaining, using at least one image sensor of an electronic device, a first image frame and multiple second image frames of a scene. Each of the second image frames has an exposure time different from an exposure time of the first image frame. The method also includes generating, using at least one processor, blur kernels indicating a motion direction of the first image frame using an optical flow network. The method further includes refining, using the at least one processor, the blur kernels using a convolutional neural network. In addition, the method includes generating, using the at least one processor, a target image frame of the scene using the refined blur kernels and occlusion masks for the second image frames.

What is claimed is: 
     
       1. A method comprising:
 obtaining, using at least one image sensor of an electronic device, a first image frame and multiple second image frames of a scene, each of the second image frames having an exposure time different from an exposure time of the first image frame; 
 generating, using at least one processor, blur kernels indicating a motion direction of the first image frame using an optical flow network, wherein the optical flow network determines an optical flow between the multiple second image frames in order to generate the blur kernels for the first image frame; 
 refining, using the at least one processor, the blur kernels using a convolutional neural network; and 
 generating, using the at least one processor, a target image frame of the scene using the refined blur kernels and occlusion masks for the second image frames. 
 
     
     
       2. The method of  claim 1 , wherein generating the target image frame of the scene comprises:
 performing an occlusion-aware deconvolution operation using the refined blur kernels and the occlusion masks; and 
 rendering a static background of the target image frame. 
 
     
     
       3. The method of  claim 1 , wherein the exposure time of each second image frame is shorter than the exposure time of the first image frame. 
     
     
       4. The method of  claim 1 , wherein generating the blur kernels comprises interpolating the optical flow at multiple pixel locations. 
     
     
       5. The method of  claim 1 , further comprising:
 estimating the occlusion masks for the second image frames before generating the target image frame of the scene. 
 
     
     
       6. The method of  claim 1 , wherein the convolutional neural network comprises multiple convolution layers and multiple transposed convolutional layers. 
     
     
       7. An apparatus comprising:
 at least one image sensor; and 
 at least one processing device configured to:
 obtain a first image frame and multiple second image frames of a scene using the at least one image sensor, each of the second image frames having an exposure time different from an exposure time of the first image frame; 
 generate blur kernels indicating a motion direction of the first image frame using an optical flow network, wherein the optical flow network is configured to determine an optical flow between the multiple second image frames in order to generate the blur kernels for the first image frame; 
 refine the blur kernels using a convolutional neural network; and 
 generate a target image frame of the scene using the refined blur kernels and occlusion masks for the second image frames. 
 
 
     
     
       8. The apparatus of  claim 7 , wherein, to generate the target image frame of the scene, the at least one processing device is configured to:
 perform an occlusion-aware deconvolution operation using the refined blur kernels and the occlusion masks; and 
 render a static background of the target image frame. 
 
     
     
       9. The apparatus of  claim 7 , wherein the exposure time of each second image frame is shorter than the exposure time of the first image frame. 
     
     
       10. The apparatus of  claim 7 , wherein, to generate the blur kernels, the at least one processing device is configured to interpolate the optical flow at multiple pixel locations. 
     
     
       11. The apparatus of  claim 7 , wherein the at least one processing device is configured to estimate the occlusion masks for the second image frames prior to generation of the target image frame of the scene. 
     
     
       12. The apparatus of  claim 7 , wherein the convolutional neural network comprises multiple convolution layers and multiple transposed convolutional layers. 
     
     
       13. A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:
 obtain a first image frame and multiple second image frames of a scene using at least one image sensor, each of the second image frames having an exposure time different from an exposure time of the first image frame; 
 generate blur kernels indicating a motion direction of the first image frame using an optical flow network, wherein the optical flow network is configured to determine an optical flow between the multiple second image frames in order to generate the blur kernels for the first image frame; 
 refine the blur kernels using a convolutional neural network; and 
 generate a target image frame of the scene using the refined blur kernels and occlusion masks for the second image frames. 
 
     
     
       14. The non-transitory machine-readable medium of  claim 13 , wherein the instructions that when executed cause the at least one processor to generate the target image frame of the scene comprise:
 instructions that when executed cause the at least one processor to:
 perform an occlusion-aware deconvolution operation using the refined blur kernels and the occlusion masks; and 
 render a static background of the target image frame. 
 
 
     
     
       15. The non-transitory machine-readable medium of  claim 13 , wherein the exposure time of each second image frame is shorter than the exposure time of the first image frame. 
     
     
       16. The non-transitory machine-readable medium of  claim 13 , wherein the instructions that when executed cause the at least one processor to generate the blur kernels comprise:
 instructions that when executed cause the at least one processor to interpolate the optical flow at multiple pixel locations. 
 
     
     
       17. The non-transitory machine-readable medium of  claim 13 , wherein the instructions when executed cause the at least one processor to estimate the occlusion masks for the second image frames prior to generation of the target image frame of the scene. 
     
     
       18. The non-transitory machine-readable medium of  claim 13 , wherein the convolutional neural network comprises multiple convolution layers and multiple transposed convolutional layers. 
     
     
       19. The method of  claim 6 , wherein the convolutional neural network is trained using reblur loss by enforcing a ground-truth sharp image, when convolved with an estimated kernel, to be the same as a blur input. 
     
     
       20. The apparatus of  claim 12 , wherein the convolutional neural network is trained using reblur loss by enforcing a ground-truth sharp image, when convolved with an estimated kernel, to be the same as a blur input.