Image processing apparatus, image capture apparatus, and control method for adding an effect of a virtual light source to a subject

Abstract

With respect to a subject included in an image, the illuminating condition by an ambient light source in an environment where the image was captured is estimated, and based on the estimation result, the effect of a virtual light source that was non-existent at the time of image capture is computed. More specifically, the effect of the virtual light source is computed using an illumination direction of the virtual light source and the reflective characteristics of the subject illuminated by the virtual light source, which have been determined based on the estimation result, and an image derived from addition of the effect of the virtual light source is output.

What is claimed is: 
     
       1. An image processing apparatus, comprising:
 a processor; and 
 a memory including instructions that, when executed by the processor, cause the processor to function as: 
 an obtainment unit configured to obtain an image derived from image capture; 
 a computation unit configured to compute an effect of a virtual light source on a subject included in the image obtained by the obtainment unit, the virtual light source being non-existent at the time of the image capture; and 
 an output unit configured to output an image derived from addition of the effect of the virtual light source to the subject based on a result of the computation by the computation unit, wherein 
 the computation unit includes: 
 an estimation unit configured to, based on the obtained image, estimate an illuminating condition by an ambient light source in an environment where the image was captured; 
 a determination unit configured to, based on a result of the estimation by the estimation unit, determine an illumination direction of the virtual light source and reflective characteristics of the subject illuminated by the virtual light source; and 
 a processing unit configured to compute the effect of the virtual light source based on the illumination direction of the virtual light source and the reflective characteristics of the subject determined by the determination unit. 
 
     
     
       2. The image processing apparatus according to  claim 1 , wherein
 the illuminating condition by the ambient light source includes at least an illumination direction of the ambient light source. 
 
     
     
       3. The image processing apparatus according to  claim 2 , wherein
 the illumination direction of the ambient light source is estimated based on a luminance distribution in a main subject region of the obtained image. 
 
     
     
       4. The image processing apparatus according to  claim 1 , wherein
 the illuminating condition by the ambient light source further includes a diffusion degree of light emitted by the ambient light source. 
 
     
     
       5. The image processing apparatus according to  claim 4 , wherein
 the diffusion degree of the light emitted by the ambient light source is estimated based on a contrast in a main subject region of the obtained image. 
 
     
     
       6. The image processing apparatus according to claim  1 , wherein
 the estimation unit estimates the illuminating condition by the ambient light source based on information of a shooting mode set at the time of the image capture of the obtained image. 
 
     
     
       7. The image processing apparatus according to  claim 1 , wherein
 the reflective characteristics of the subject include a diffuse reflectance and a specular reflectance of the subject under light incident from the virtual light source. 
 
     
     
       8. The image processing apparatus according to  claim 7 , wherein
 the diffuse reflectance and the specular reflectance of the subject are determined based on a category of the subject. 
 
     
     
       9. The image processing apparatus according to  claim 1 , wherein
 the computation unit further includes a specifying unit configured to specify categories of a plurality of subjects included in a main subject region of the obtained image, and 
 the computation unit defines virtual light sources that differ among the categories of the subjects specified by the specifying unit, and computes an effect of each virtual light source on a corresponding one of the subjects. 
 
     
     
       10. An image processing apparatus, comprising:
 a processor; and 
 a memory including instructions that, when executed by the processor, cause the processor to function as: 
 an obtainment unit configured to obtain an image derived from image capture; 
 a computation unit configured to compute an effect of a virtual light source on a subject included in the image obtained by the obtainment unit, the virtual light source being non-existent at the time of the image capture; and 
 an output unit configured to output an image derived from addition of the effect of the virtual light source to the subject based on a result of the computation by the computation unit, wherein 
 the computation unit includes: 
 a determination unit configured to, based on a shooting scene of the obtained image or a type of the subject, determine an illumination direction of the virtual light source and reflective characteristics of the subject illuminated by the virtual light source; and 
 a processing unit configured to compute the effect of the virtual light source based on the illumination direction of the virtual light source and the reflective characteristics of the subject determined by the determination unit. 
 
     
     
       11. An image capture apparatus, comprising:
 a processor; and 
 a memory including instructions that, when executed by the processor, cause the processor to function as: 
 an image capture unit configured to generate an image through image capture; and 
 an image processing apparatus that obtains the image generated by the image capture unit and outputs an image derived from addition of an effect of a virtual light source, wherein the image processing apparatus comprising: 
 an obtainment unit configured to obtain an image derived from image capture; 
 a computation unit configured to compute the effect of the virtual light source on a subject included in the image obtained by the obtainment unit, the virtual light source being non-existent at the time of the image capture; and 
 an output unit configured to output an image derived from addition of the effect of the virtual light source to the subject based on a result of the computation by the computation unit, and wherein the computation unit includes: 
 an estimation unit configured to, based on the obtained image, estimate an illuminating condition by an ambient light source in an environment where the image was captured; 
 a determination unit configured to, based on a result of the estimation by the estimation unit, determine an illumination direction of the virtual light source and reflective characteristics of the subject illuminated by the virtual light source; and 
 a processing unit configured to compute the effect of the virtual light source based on the illumination direction of the virtual light source and the reflective characteristics of the subject determined by the determination unit. 
 
     
     
       12. An image processing apparatus, comprising:
 a processor; and 
 a memory including instructions that, when executed by the processor, cause the processor to function as: 
 an evaluation unit configured to evaluate a state of light illumination of a subject included in a shot image, the light illumination being performed by an ambient light source at the time of shooting; and 
 a generation unit configured to generate a corrected image by applying, to the shot image, image processing for adding an effect of light illumination performed by a virtual light source based on the evaluation by the evaluation unit, the virtual light source being non-existent at the time of the shooting, wherein 
 in applying the image processing, the generation unit controls an arrangement of the virtual light source and reflective characteristics of the subject under the light illumination performed by the virtual light source based on the evaluation by the evaluation unit. 
 
     
     
       13. The image processing apparatus according to  claim 12 , wherein
 the reflective characteristics of the subject include a diffuse reflectance and a specular reflectance of the subject under light incident from the virtual light source. 
 
     
     
       14. The image processing apparatus according to  claim 13 , wherein
 the diffuse reflectance and the specular reflectance of the subject are determined based on a category of the subject. 
 
     
     
       15. A control method for an image processing apparatus, the control method comprising:
 obtaining an image derived from image capture; 
 computing an effect of a virtual light source on a subject included in the obtained image, the virtual light source being non-existent at the time of the image capture; and 
 outputting an image derived from addition of the effect of the virtual light source to the subject based on a result of the computing, wherein 
 the computing includes 
 based on the obtained image, estimating an illuminating condition by an ambient light source in an environment where the image was captured; 
 based on a result of the estimation in the estimating, determining an illumination direction of the virtual light source and reflective characteristics of the subject illuminated by the virtual light source; and 
 executing processing for computing the effect of the virtual light source based on the determined illumination direction of the virtual light source and the determined reflective characteristics of the subject. 
 
     
     
       16. A control method for an image processing apparatus, the control method comprising:
 evaluating a state of light illumination of a subject included in a shot image, the light illumination being performed by an ambient light source at the time of shooting; and 
 generating a corrected image by applying, to the shot image, image processing for adding an effect of light illumination performed by a virtual light source based on the evaluation in the evaluating, the virtual light source being non-existent at the time of the shooting, wherein 
 in applying the image processing in the generating, an arrangement of the virtual light source and reflective characteristics of the subject under the light illumination performed by the virtual light source are controlled based on the evaluation in the evaluating.