Data pre-processing for cross sensor automatic white balance

Abstract

Learning-based color correction (e.g., auto while balance (AWB)) procedures may be trained based on datasets from different sensors using a pre-processing procedure. Each input pixel may be converted into a sensor-independent representation through multiplication by a sensor-specific color conversion function (e.g., a 3×3 matrix). The sensor-specific color conversion function (e.g., the 3×3 matrix) may be obtained based on a sensor type. For example, the sensor-specific color conversion function, such as a 3×3 matrix, may be obtained by a corresponding sensor calibration procedure performed using laboratory images of a color checker chart subject to standard illuminants. Parameters of the sensor-specific color conversion function may be optimized in a chromaticity space. For instance, a sensor-specific 3×3 matrix for color conversion may be optimized using a distance in the chromaticity space between calibration data (e.g., calibration configurations) and sensor-independent targets (e.g., a target sensor-independent representation for each calibration configuration).

What is claimed is: 
     
         1 . A method for image processing, comprising:
 identifying image data for an image, wherein the image data comprises a sensor type;   generating a sensor-independent representation of the image using a sensor-specific color conversion function corresponding to the sensor type, wherein parameters of the sensor-specific color conversion function have been optimized in a chromaticity space; and   producing a sensor-independent illuminance estimate for the image based on the sensor-independent representation.   
     
     
         2 . The method of  claim 1 , further comprising:
 capturing the image using a sensor corresponding to the sensor type.   
     
     
         3 . The method of  claim 2 , further comprising:
 determining a spectral sensitivity of the sensor; and   determining the sensor type based on the spectral sensitivity.   
     
     
         4 . The method of  claim 1 , further comprising:
 applying an inverse of the sensor-specific color conversion function to the sensor-independent illuminance estimate to produce a sensor-specific illuminance estimate; and   performing color correction on the image based on the sensor-specific illuminance estimate.   
     
     
         5 . The method of  claim 4 , further comprising:
 displaying a white-balanced version of the image to a user based on the color correction.   
     
     
         6 . The method of  claim 1 , further comprising:
 determining a target sensor-independent representation for each of a plurality of calibration configurations by optimizing in the chromaticity space; and   determining the sensor-specific color conversion function for the sensor type based on the target sensor-independent representation for each of the plurality of calibration configurations.   
     
     
         7 . The method of  claim 1 , wherein:
 the image data comprises a 3-channel image vector, and the sensor-specific color conversion function comprises a 3×3 matrix.   
     
     
         8 . A method for image processing, comprising:
 determining a target sensor-independent representation for each of a plurality of calibration configurations by optimizing in a chromaticity space;   determining a sensor-specific color conversion function for each of a plurality of sensor types by optimizing in the chromaticity space based on the target sensor-independent representation for each of the plurality of calibration configurations; and   performing white balancing for an image captured with a sensor type from the plurality of sensor types based on the corresponding sensor-specific color conversion function.   
     
     
         9 . The method of  claim 8 , further comprising:
 identifying a plurality of configuration images, wherein each of the plurality of configuration images comprises a configuration sensor type from the plurality of sensor types and a calibration configuration from the plurality of calibration configurations; and   iterating between a first step and a second step to produce a reverse color conversion function for each of the plurality of sensor types and a target sensor-independent representation for each of the plurality of calibration configurations,   wherein the first step optimizes the reverse color conversion function to minimize a first distance in the chromaticity space between each of a plurality of calibration images and a product of the reverse color conversion function and a previous iteration of the target sensor-independent representation, wherein the first distance is aggregated across all of the calibration configurations and the first optimization is performed for each of the plurality of sensor types independently; and   wherein the second step optimizes target sensor-independent representation to minimize a second distance in the chromaticity space between each of the plurality of calibration images and a product the optimized reverse color conversion function from the first step and target sensor-independent representation, wherein the second distance is aggregated across all of the plurality of sensor types and is performed for each of the plurality of calibration configurations independently.   
     
     
         10 . The method of  claim 9 , further comprising:
 initializing the target sensor-independent representation for each of the plurality of calibration configurations based on a reference illuminance.   
     
     
         11 . The method of  claim 8 , further comprising:
 minimizing a distance between a chromaticity-space representation of a product of a color conversion function and a plurality of calibration images corresponding to a corresponding sensor type and the plurality of calibration configurations to a chromaticity space representation of each of the plurality of calibration configurations, wherein the target sensor-independent representation for each of the plurality of calibration configurations is based on the minimizing.   
     
     
         12 . The method of  claim 8 , wherein:
 each of the plurality of calibration configurations comprises a reflectance and an illuminance.   
     
     
         13 . The method of  claim 8 , wherein:
 the plurality of calibration configurations represents an evenly spaced covering of a color space.   
     
     
         14 . The method of  claim 8 , further comprising:
 identifying image data comprising a 3-channel image vector, wherein each of the sensor-specific color conversion functions comprises a 3×3 matrix.   
     
     
         15 . The method of  claim 8 , further comprising:
 performing pre-processing on the image using the sensor-specific color conversion function to produce a sensor-independent representation of the image;   generating a sensor-independent illuminance estimate based on the sensor-independent representation of the image; and   performing post-processing on the sensor-independent illuminance estimate using an inverse of the sensor-specific color conversion function, wherein the white balancing is performed based on the post-processing.   
     
     
         16 . The method of  claim 8 , further comprising:
 identifying training data for an illuminance estimation model, wherein the training data comprises a plurality of input images, and wherein each of the input images is associated with an input sensor type and a ground truth illuminance;   applying the sensor-specific color conversion function for the input sensor type to each of the input images to produce a sensor-independent representation for an input image of the plurality of input images;   applying the illuminance estimation model to generate a sensor-independent illuminance estimate for the input image;   applying the sensor-specific color conversion function for the input sensor type to the ground truth illuminance to produce a sensor-independent illuminance;   comparing the sensor-independent illuminance estimate to the sensor-independent illuminance; and   updating the illuminance estimation model based on the comparison, wherein the white balancing is performed based on the illuminance estimation model.   
     
     
         17 . An apparatus for image processing, comprising:
 a sensor configured to capture image data, wherein the sensor has a spectral sensitivity corresponding to a sensor type;   a pre-processing component configured to generate a sensor-independent representation of the image using a sensor-specific color conversion function corresponding to the sensor type, wherein parameters of the sensor-specific color conversion function have been optimized in a chromaticity space;   an illuminance estimation component configured to produce a sensor-independent illuminance estimate based on the sensor-independent representation; and   a post-processing component configured to generate a sensor-specific illuminance estimate based on the sensor-independent illuminance estimate using an inverse of the sensor-specific color conversion function.   
     
     
         18 . The apparatus of  claim 17 , further comprising:
 a color correction component configured to perform color correction on the image based on the sensor-specific illuminance estimate.   
     
     
         19 . The apparatus of  claim 18 , further comprising:
 a display configured to display the image to a user based on the color correction.   
     
     
         20 . The apparatus of  claim 17 , further comprising:
 a calibration component configured to optimize the parameters of the sensor-specific color conversion function in the chromaticity space.