Image fusion system and method

Abstract

A contrast-based image fusion system and method of processing multiple images to form a processed or fused image including regions selected from one or more images. Images are divided into image regions. Portions of the images are filtered if necessary. A contrast map is generated for each image via a convolution kernel resulting in a contrast map with contrast values for each image region. Contrast values are compared and image regions are selected based on a selection criteria or process such as greater or maximum contrast. The selected image regions form the fused image. If necessary, the luminance of one or more portions of the fused image is adjusted. One sensor is selected as a reference sensor, and an average intensity of each region of the reference sensor image is determined across the reference sensor image. The intensity of one or more regions in the final image is adjusted by combining the determined average luminance values and intensity values of the final image.

1. A method of forming a processed image using a plurality of images, each image generated by a respective sensor, comprising:
 dividing each image into a plurality of image regions;  
 generating a contrast map for each image, each contrast map including a contrast value for each image region;  
 applying a selection process to said contrast value for selecting an image region for use in said processed image; and  
 forming said processed image with the selected image regions,  
 wherein contrast values of contrast maps of respective first, second, and third sensors are compared together to form said processed image, the method further comprising:  
 identifying contrast values from first and second sensor images to form an intermediate contrast map;  
 wherein applying the selection process comprises applying a selection process to the contrast values of the intermediate contrast map and contrast values of a contrast map of a third sensor image.  
 
     
     
       2. The method of  claim 1 , wherein dividing the images into the plurality of image regions further comprises dividing each image on a pixel-by-pixel basis, into blocks of pixels, or into arbitrary shaped regions. 
     
     
       3. The method of  claim 1 , wherein each sensor detects a different wavelength. 
     
     
       4. The method of  claim 1 , wherein the plurality of sensors includes an infrared (IR) sensor and a radar sensor. 
     
     
       5. The method of  claim 1 , wherein the plurality of sensors includes an infrared (IF) and an ultraviolet (UV) sensor. 
     
     
       6. The method of  claim 1 , wherein the plurality of sensors includes a radar sensor and an ultraviolet (UV) sensor. 
     
     
       7. The method of  claim 1 , wherein the plurality of images are generated by two or more infrared (IR) sensors, each IR sensor detecting a different wavelength. 
     
     
       8. The method of  claim 1 , wherein applying the selection process includes comparing competing contrast values of two corresponding image regions from two respective images. 
     
     
       9. The method of  claim 8 , wherein said selection process operates to select the greater of the competing contrast values. 
     
     
       10. The method of  claim 8 , wherein comparing competing contrast values further comprises comparing corresponding contrast values of overlapping image regions. 
     
     
       11. The method of  claim 1 , wherein the first and second sensors are infrared (IR) sensors and the third sensor is a radar sensor. 
     
     
       12. The method of  claim 1 , wherein a sensor image displays a view from a moving vehicle. 
     
     
       13. The method of  claim 12 , wherein the moving vehicle is an aircraft, a watercraft, an automobile, or a train. 
     
     
       14. The method of  claim 1 , further comprising adjusting an intensity of one or more regions of said processed image. 
     
     
       15. The method of  claim 14 , further comprising weighting the degree of intensity adjustment. 
     
     
       16. The method of  claim 14 , wherein adjusting the intensity further comprises adjusting the intensity across said processed image. 
     
     
       17. The method of  claim 14 , wherein adjusting the intensity across said processed image further comprises:
 selecting one sensor as a reference sensor;  
 determining at least one average intensity value for each region of the reference sensor image; and  
 adjusting the intensity of one or more regions in said processed image by combining the determined average intensity values and intensity values of said processed image.  
 
     
     
       18. The method of  claim 17 , wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the radar sensor. 
     
     
       19. The method of  claim 17 , wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the infrared (IR) sensor. 
     
     
       20. The method of  claim 17 , wherein adjusting the intensity of one or more regions in said processed image further comprises adjusting the intensity of a line in said processed image corresponding to a line in the reference sensor image for which the average intensity was determined. 
     
     
       21. The method of  claim 20 , wherein adjusting the intensity of one or more lines in said processed image further comprises adjusting the intensity of a line in said processed image that is adjacent to a line in said processed image corresponding to the same line in the reference sensor image for which the average intensity was determined. 
     
     
       22. The method of  claim 1 , before generating the contrast map for each image, further comprising filtering regions of one or more images. 
     
     
       23. The method of  claim 22 , wherein filtering further comprises spatially filtering regions of each image by weighting selected image regions. 
     
     
       24. The method of  claim 23 , wherein one sensor comprises a radar sensor, and wherein spatial filtering is performed by filtering image regions above a radar horizon. 
     
     
       25. The method of  claim 1 , wherein each of the sensors detects substantially the same scene such that each of the images is associated with substantially the same scene. 
     
     
       26. The method of  claim 25 , wherein each of the sensors detects a range of wavelengths that is different than a range of wavelengths detected by the other sensors, such that each of the images is associated with a different range of wavelengths. 
     
     
       27. A method of forming a processed image using a plurality of images, each image generated by a respective sensor, comprising:
 dividing each image into a plurality of image regions;  
 generating a contrast map for each image, each contrast map including a contrast value for each image region;  
 applying a selection process to said contrast value for selecting an image region for use in said processed image; and  
 forming said processed image with the selected image regions;  
 wherein generating the contrast map further comprises performing a convolution to determine the contrast value of the contrast map;  
 wherein performing the convolution further comprises performing the convolution with a Kernel Kc, 
 wherein [{Kc*S 1 (x,y), Kc*S 2 (x,y)}] represents the convolution; 
         K   c     =                  -     1     2   ⁢     2                 -     1   2             -     1     2   ⁢     2                     -     1   2             2   ⁢       1   +     2         2               -     1   2                 -     1     2   ⁢     2                 -     1   2             -     1     2   ⁢     2                      ≅                -   0.354           -   0.500           -   0.354               -   0.500         3.414         -   0.500               -   0.354           -   0.500           -   0.354                      
 
 S 1  represents image regions of a first image;  
 S 2  represents image regions of a second image; and  
 (x,y) represent spatial coordinates of the images.  
 
 
     
     
       28. The method of  claim 27 , wherein dividing the images into the plurality of image regions further comprises dividing each image on a pixel-by-pixel basis, into blocks of pixels, or into arbitrary shaped regions. 
     
     
       29. The method of  claim 27 , wherein each sensor detects a different wavelength. 
     
     
       30. The method of  claim 27 , wherein applying the selection process includes comparing competing contrast values of two corresponding image regions from two respective images. 
     
     
       31. The method of  claim 30 , wherein said selection process operates to select the greater of the competing contrast values. 
     
     
       32. The method of  claim 30 , wherein comparing competing contrast values further comprises comparing corresponding contrast values of overlapping image regions. 
     
     
       33. A system for combining a plurality of images to form a final image, comprising:
 a plurality of sensors that generate respective images;  
 a processor configured to 
 divide each image into a plurality of image regions,  
 generate a contrast map for each image, each contrast map including a contrast value for each image region,  
 apply a selection criterion to said contrast value for selecting an image region for use in said processed image, and  
 form said processed image with the selected image regions;  
 
 wherein contrast values of contrast maps of a respective first, second, and third sensors are compared together to form the final image;  
 wherein the processor is further configured to identify contrast values from first and second sensor images to form an intermediate contrast map; and  
 wherein the processor applies the selection criterion by applying a selection process to the contrast values of the intermediate contrast map and contrast values of a contrast map of a third sensor image.  
 
     
     
       34. The system of  claim 33 , wherein the processor is configured to divide each image into individual pixels, into blocks of pixels, or into arbitrary shaped regions. 
     
     
       35. The system of  claim 33 , wherein each sensor detects a different wavelength. 
     
     
       36. The system of  claim 33 , wherein the plurality of sensors includes an infrared (IR) sensor and a radar sensor. 
     
     
       37. The system of  claim 33 , wherein the plurality of sensors includes an infrared (IR) and an ultraviolet (UV) sensor. 
     
     
       38. The system of  claim 33 , wherein the plurality of sensors includes a radar sensor and an ultraviolet (UV) sensor. 
     
     
       39. The system of  claim 33 , wherein the plurality of sensors includes two or more infrared (IR) sensors, each IR sensor detecting a different wavelength. 
     
     
       40. The system of  claim 33 , wherein the processor is further configured to compare competing contrast values of two corresponding image regions from two respective images. 
     
     
       41. The system of  claim 40 , wherein the processor is further configured to select the greater of the competing contrast values. 
     
     
       42. The system of  claim 40 , wherein the processor is configured to compare corresponding contrast values of overlapping image regions. 
     
     
       43. The system of  claim 33 , wherein the first and second sensors are infrared (IR) sensors and the third sensor is a radar sensor. 
     
     
       44. The system of  claim 33 , wherein a sensor image displays a view from a moving vehicle. 
     
     
       45. The system of  claim 44 , wherein the moving vehicle comprises an aircraft, a watercraft, an automobile, or a train. 
     
     
       46. The system of  claim 33 , wherein the processor is further configured to adjust an intensity of one or more regions of said processed image. 
     
     
       47. The system of  claim 46 , wherein the processor is configured to adjust the intensity across said processed image. 
     
     
       48. The system of  claim 47 , wherein the processor is configured to weight the degree of intensity adjustment. 
     
     
       49. The system of  claim 47 , wherein the processor is further configured to
 select one sensor as a reference sensor,  
 determine at least one average intensity value for each region of the reference sensor image across the reference sensor image, and  
 adjust the intensity of one or more regions in said processed image by combining the determined average intensity values and intensity values of said processed image.  
 
     
     
       50. The system of  claim 49 , wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the radar sensor. 
     
     
       51. The system of  claim 49 , wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the infrared (IR) sensor. 
     
     
       52. The system of  claim 49 , wherein the processor is configured to adjust the intensity of a line in said processed image corresponding to a line in the reference sensor image for which the average intensity was determined. 
     
     
       53. The system of  claim 49 , wherein the processor is configured to adjust the intensity of a line in said processed image that is adjacent to a line in said processed image corresponding to the same line in the reference sensor image for which the average intensity was determined. 
     
     
       54. The system of  claim 33 , wherein the processor is configured to filter one or more image regions. 
     
     
       55. The system of  claim 54 , wherein the processor is configured to filter the one or more image regions by weighting selected image regions. 
     
     
       56. The system of  claim 54 , wherein one sensor comprises a radar sensor, and wherein the processor is further configured to spatially filter image regions above a radar horizon. 
     
     
       57. The system of  claim 33 , wherein each of the sensors detects substantially the same scene such that each of the respective images is associated with substantially the same scene. 
     
     
       58. The system of  claim 33 , wherein each of the sensors detects a range of wavelengths that is different than a range of wavelengths detected the other sensors, such that each of the respective images is associated with a different range of wavelengths. 
     
     
       59. A system for combining a plurality of images to form a final image, comprising:
 a plurality of sensors that generate respective images; and  
 a processor configured to 
 divide each image into a plurality of image regions,  
 generate a contrast map for each image, each contrast map including a contrast value for each image region,  
 apply a selection criterion to said contrast value for selecting an image region for use in said processed image, and  
 form said processed image with the selected image regions;  
 
 wherein the processor is configured to generate the contrast map by performing a convolution to determine the contrast value of the contrast map; and  
 wherein the processor is configured to perform the convolution with a Kernel Kc, 
 wherein [{Kc*S 1 (x,y), Kc*S 2 (x,y)}] represents the convolution; 
         K   c     =                  -     1     2   ⁢     2                 -     1   2             -     1     2   ⁢     2                     -     1   2             2   ⁢       1   +     2         2               -     1   2                 -     1     2   ⁢     2                 -     1   2             -     1     2   ⁢     2                      ≅                -   0.354           -   0.500           -   0.354               -   0.500         3.414         -   0.500               -   0.354           -   0.500           -   0.354                      
 
 S 1  represents image regions of a first image;  
 S 2  represents image regions of a second image; and  
 (x,y) represent spatial coordinates of the images.  
 
 
     
     
       60. The system of  claim 59 , wherein the processor is configured to divide each image into individual pixels, into blocks of pixels, or into arbitrary shaped regions. 
     
     
       61. The system of  claim 59 , wherein each sensor detects a different wavelength. 
     
     
       62. The system of  claim 59 , wherein the processor is further configured to compare competing contrast values of two corresponding image regions from two respective images. 
     
     
       63. The system of  claim 62 , wherein the processor is further configured to select the greater of the competing contrast values. 
     
     
       64. The system of  claim 62 , wherein the processor is configured to compare corresponding contrast values of overlapping image regions. 
     
     
       65. A method of forming a processed image using a plurality of images, each image generated by a respective sensor, comprising:
   dividing each image into a plurality of image regions;        generating a contrast map for each image, each contrast map including a contrast value for each image region;        applying a selection process to said contrast value for selecting an image region for use in said processed image; and        forming said processed image with the selected image regions        wherein dividing the images into the plurality of image regions further comprises dividing each image on a pixel - by - pixel basis, into blocks of pixels, or into arbitrary shaped regions, and        wherein applying the selection process includes comparing competing contrast values of corresponding image regions from respective images.     
     
     
       66. The method of  claim 65 , wherein:
   each sensor detects a range of wavelengths that is different than a range of wavelengths detected by the other sensors such that each of the images is associated with a different range of wavelengths;        dividing each image comprises defining sets of corresponding image regions by dividing each image into a plurality of image regions each of which is substantially aligned with and substantially overlaps with one image region in each of the other images; and        selecting an image region for use in the processed image comprises comparing, for each set of corresponding image regions, the contrast values for each image region in the set and selecting one image region from the set based on the comparing.     
     
     
       67. The method of  claim 65 , wherein the sensors are fixed to a moving vehicle, wherein each sensor detects substantially the same field of view from the moving vehicle such that the images represent substantially the same field of view. 
     
     
       68. The method of  claim 65 , wherein the plurality of sensors includes at least two of an infrared sensor, an ultraviolet sensor, and a radio frequency sensor. 
     
     
       69. The method of  claim 68 , wherein the radio frequency sensor comprises an imaging radar sensor. 
     
     
       70. The method of  claim 65 , wherein each sensor is one of an infrared sensor, an ultraviolet sensor, or a radio frequency sensor. 
     
     
       71. The method of  claim 65 , wherein each sensor detects a range of wavelengths that is different than a range of wavelengths detected by the other sensors. 
     
     
       72. The method of  claim 65 , comprising adjusting an intensity value associated with at least one region of the processed image based on an average intensity value associated with at least a portion of one of the processed image. 
     
     
       73. The method of  claim 72 , wherein adjusting the intensity value further comprises adjusting the intensity value across said processed image. 
     
     
       74. The method of  claim 73 , wherein adjusting the intensity value across said processed image further comprises:
   selecting one sensor as a reference sensor;        determining an average intensity value of each region of the reference sensor image; and        adjusting the intensity value of one or more regions in said processed image by combining the determined average intensity values and intensity values of said processed image.     
     
     
       75. The method of  claim 74 , wherein adjusting the intensity value of one or more regions in said processed image further comprises adjusting the intensity value of a line in said processed image corresponding to a line in the reference sensor image for which the average intensity value was determined. 
     
     
       76. The method of  claim 75 , wherein adjusting the intensity value of one or more lines in said processed image further comprises adjusting the intensity value of a line in said processed image that is adjacent to a line in said processed image corresponding to the same line in the reference sensor image for which the average intensity value was determined. 
     
     
       77. The method of  claim 74 , wherein a scene horizon is repositioned at an angle relative to an image horizon, further comprising:
   determining an average intensity value of the reference sensor image on a pixel - by - pixel basis; and        adjusting the intensity value of said processed image on a pixel - by - pixel basis.     
     
     
       78. The method of  claim 77 , wherein the scene horizon is repositioned due to roll, bank, yaw or pitch motions. 
     
     
       79. The method of  claim 65 , comprising filtering regions of one or more images before generating the contrast map for each image. 
     
     
       80. The method of  claim 79 , wherein filtering comprises spatially filtering regions of each image by weighting selected image regions. 
     
     
       81. The method of  claim 80 , wherein one sensor comprises a radar sensor, and wherein spatial filtering is performed by filtering image regions above a radar horizon. 
     
     
       82. The method of  claim 65 , wherein generating the contrast map further comprises performing a convolution to determine the contrast value of the contrast map. 
     
     
       83. The method of  claim 65 , wherein said selection process operates to select the greater of the competing contrast values. 
     
     
       84. The method of  claim 65 , wherein comparing competing contrast values further comprises comparing corresponding contrast values of overlapping image regions. 
     
     
       85. The method of  claim 65 , wherein a sensor image displays a view from a moving vehicle. 
     
     
       86. The method of  claim 85 , wherein the moving vehicle is an aircraft, a watercraft, an automobile, or a train. 
     
     
       87. A method of forming a processed image using a plurality of images, each image generated by a respective sensor to form a processed image, comprising:
   filtering portions of one or more images;        comparing contrast values of the images by      dividing each image into a plurality of image regions,        generating a contrast map for each image, each contrast map including contrast values for each image region of each image;        comparing contrast values in each contrast map of the image regions,        identifying maximum contrast values based on the comparison of contrast values, and        selecting image regions corresponding to the maximum contrast values,          forming the processed image with the selected image regions; and        adjusting an intensity of one or more portions of the processed image by selecting one sensor as a reference sensor,      determining at least one average intensity value for one or more regions of the reference sensor image across the reference sensor image, and        adjusting the intensity of one or more regions in the processed image by combining the determined average intensity values and intensity values of the processed image;          wherein generating the contrast map further comprises performing a convolution to determine the contrast value of the contrast map.     
     
     
       88. The method of  claim 87 , wherein performing the convolution comprises performing a convolution over two image regions that are spatially pre-registered or aligned. 
     
     
       89. The method of  claim 87 , wherein performing the convolution comprises using a kernel including values that reflect a distance metric from a center position. 
     
     
       90. The method of  claim 87 , wherein the plurality of sensors includes at least two of an infrared sensor, an ultraviolet sensor, and a radio frequency sensor. 
     
     
       91. The method of  claim 90 , wherein the radio frequency sensor comprises an imaging radar sensor. 
     
     
       92. The method of  claim 87 , wherein each sensor detects a range of wavelengths that is different than a range of wavelengths detected by the other sensors. 
     
     
       93. The method of  claim 87 , wherein applying the selection process includes comparing competing contrast values of two corresponding image regions from two respective images. 
     
     
       94. The method of  claim 87 , wherein each sensor is one of an infrared sensor, an ultraviolet sensor, or a radio frequency sensor. 
     
     
       95. The method of  claim 94 , wherein the radio frequency sensor comprises an imaging radar sensor. 
     
     
       96. The method of  claim 87 , wherein each sensor detects a range of wavelengths that is different than a range of wavelengths detected by the other sensors. 
     
     
       97. The method of  claim 87 , wherein selecting image regions includes comparing competing contrast values of two corresponding image regions from two respective images. 
     
     
       98. The method of  claim 87 , wherein filtering comprises spatially filtering regions of each image by weighting selected image regions. 
     
     
       99. The method of  claim 98 , wherein one sensor comprises a radar sensor, and wherein spatial filtering is performed by filtering image regions above a radar horizon. 
     
     
       100. A method of forming a processed image using a plurality of images, each image generated by a respective sensor, comprising:
   comparing contrast values of contrast maps of the images by      defining a plurality of sets of corresponding image regions in the plurality of images;        generating contrast maps with contrast values for the sets of corresponding image regions in the plurality of images;        identifying, for each set of corresponding image regions, one contrast value as a maximum contrast value, and          selecting image regions corresponding to the maximum contrast values,        forming a processed image using the selected image regions; and        adjusting an intensity of at least one portion of the processed image by:      determining at least one intensity value for the at least one portion of the processed image,        selecting one of the sensors as a reference sensor,        determining an average intensity value for one or more regions of an image generated by the reference sensor, and        adjusting the at least one intensity value for the at least one portion of the processed image in accordance with the determined average intensity value.       
     
     
       101. A system for combining a plurality of images to form a final image, comprising:
   a plurality of sensors that generate respective images;        a processor configured to      divide each image into a plurality of image regions,        generate a contrast map for each image, each contrast map including a contrast value for each image region,        apply a selection criterion to said contrast value for selecting an image region for use in said processed image, and        form said processed image with the selected image regions;          wherein the processor is further configured to divide the images into the plurality of image regions by dividing each image on a pixel - by - pixel basis, into blocks of pixels, or into arbitrary shaped regions, and        wherein the process is further configured to apply the selection process by comparing competing contrast values of corresponding image regions from respective images.     
     
     
       102. The system of  claim 101 , wherein each sensor detects a different wavelength. 
     
     
       103. The system of  claim 101 , wherein at least one sensor is of the following group: an infrared ( IR )  sensor, a radar sensor, and an ultraviolet  ( UV )  sensor.   
     
     
       104. The system of  claim 101 , wherein the processor is further configured to compare competing contrast values of two corresponding image regions from two respective images. 
     
     
       105. The system of  claim 101 , wherein the processor is further configured to select the greater of the competing contrast values. 
     
     
       106. The system of  claim 101 , wherein the processor is configured to compare corresponding contrast values of overlapping image regions. 
     
     
       107. The system of  claim 101 , wherein a sensor image displays a view from a moving vehicle. 
     
     
       108. The system of  claim 107 , wherein the moving vehicle comprises an aircraft, a watercraft, an automobile, or a train. 
     
     
       109. The system of  claim 101 , wherein the processor is further configured to adjust an intensity of one or more regions of said processed image. 
     
     
       110. The system of  claim 109 , wherein the processor is configured to adjust the intensity across said processed image. 
     
     
       111. The system of  claim 110 , wherein the processor is configured to weight the degree of intensity adjustment. 
     
     
       112. The system of  claim 110 , wherein the processor is further configured to select one sensor as a reference sensor, determine at least one average intensity value for each region of the reference sensor image across the reference sensor image, and adjust the intensity of one or more regions in said processed image by combining the determined average intensity values and intensity values of said processed image. 
     
     
       113. The system of  claim 112 , wherein the processor is configured to adjust the intensity of a line in said processed image corresponding to a line in the reference sensor image for which the average intensity was determined. 
     
     
       114. The system of  claim 112 , wherein the processor is configured to adjust the intensity of a line in said processed image that is adjacent to a line in said processed image corresponding to the same line in the reference sensor image for which the average intensity was determined. 
     
     
       115. The system of  claim 101 , wherein the processor is configured to filter one or more image regions. 
     
     
       116. The system of  claim 115 , wherein the processor is configured to filter the one or more image regions by weighting selected image regions. 
     
     
       117. The system of  claim 115 , wherein one sensor comprises a radar sensor, and wherein the processor is further configured to spatially filter image regions above a radar horizon. 
     
     
       118. The system of  claim 101 , wherein the processor is configured to generate the contrast map by performing a convolution to determine the contrast value of the contrast map. 
     
     
       119. A system for forming a processed image using a plurality of images, comprising:
   a first sensor that generates a first image;        a second sensor that generates a second image,        wherein the first and second images are divided into a plurality of image regions;        a processor configured to      filter one or more portions of one or more images;        compare contrast values of the images by      dividing each image into a plurality of image regions,        generating a contrast map for each image, each contrast map including contrast values for each image region of each image,        comparing contrast values in each contrast map of the image regions,        identifying maximum contrast values,        selecting image regions corresponding to the maximum contrast values, and        forming the processed image with the selected image regions; and          adjust an intensity of one or more regions of the final image by      selecting one sensor as a reference sensor,        determining at least one average intensity value for each region of the reference sensor image across the reference sensor image, and        adjusting the intensity of one or more regions in said processed image by combining the determined average intensity values and intensity values of the final image;            wherein the processor is further configured to divide the images into the plurality of image regions by dividing each image on a pixel - by - pixel basis, into blocks of pixels, or into arbitrary shaped regions, and        wherein the processor is further configured to apply the selection process by comparing competing contrast values of corresponding image regions from respective images.