Color interpolator and horizontal/vertical edge enhancer using two line buffer and alternating even/odd filters for digital camera

Abstract

A digital-camera processor receives a stream of mono-color pixels in a Bayer pattern from a sensor. Two lines of the pattern are stored in a 2-line buffer. Red, Blue, and Green interpolators receive a 3×3 array of pixels from the 2-line buffer. The interpolators generate missing color values by interpolation. For green, horizontal interpolation is performed for odd lines, while vertical interpolation is performed for even lines. Horizontal and vertical interpolation is thus alternated with alternate lines. Edge detection is performed at the same time as interpolation, on the green pixels from the 2-line buffer. An edge-detection filter is multiplied by the green pixels in the 3×3 array from the 2-line buffer. Different edge-detection filters are used for odd and even lines. These filters are modified to detect edges running perpendicular to the direction of the green interpolation filter. Edges in the same direction as the interpolation filter are ignored. Thus blurring caused by the green interpolation does not affect edge detection. The result of the edge-detection filter is compared to two different threshold values, one for green and one for red and blue. When an edge is detected, an edge enhancer is activated. The edge enhancer adds a scaled factor to the interpolated R, G, or B values to sharpen the detected edge. The line buffer stores only 2 full lines of pixels and no full-frame buffer is needed.

1. An alternating-line color interpolator and edge detector comprising:
 an input for receiving a stream of mono-color pixels from an image sensor; 
 a 2-line buffer, coupled to receive pixels from the input, for storing a full line of pixels and portions of 2 other lines of pixels, the 2-line buffer outputting a 3×3 region of pixels having 3 rows and 3 columns; 
 a red interpolator, receiving red pixels in the 3×3 region from the 2-line buffer, for averaging the red pixels within the 3×3 region to generate an interpolated red pixel for a center location of the 3×3 region; 
 a blue interpolator, receiving blue pixels in the 3×3 region from the 2-line buffer, for averaging the blue pixels within the 3×3 region to generate an interpolated blue pixel for a center location of the 3×3 region; 
 a green interpolator, receiving green pixels in the 3×3 region from the 2-line buffer, for averaging the green pixels within the 3×3 region to generate an interpolated green pixel for a center location of the 3×3 region; 
 wherein for odd lines the green interpolator averages only green pixels in a middle column of the 3×3 region when the center location is on an odd line, the green interpolator ignoring green pixels in a left column to the left of the middle column and in a right column to the right of the middle column for odd lines; 
 wherein for even lines the green interpolator averages only green pixels in a middle row of the 3×3 region when the center location is on an even line, the green interpolator ignoring green pixels in an upper row above the middle row and in a lower row below the middle row for even lines; and 
 an edge detector, receiving green pixels in the 3×3 region from the 2-line buffer, for multiplying each green pixel by a filter coefficient and for summing results to generate an edge value; 
 wherein for odd lines, the filter coefficients for green pixels in a middle column have an opposite sign as filter coefficients for green pixels in the right and left column, 
 whereby edges crossing the middle column are magnified more than vertical edges within the middle column for odd lines; 
 wherein for even lines, the filter coefficients for green pixels in a middle row have an opposite sign as filter coefficients for green pixels in the upper and lower row, 
 whereby edges crossing the middle row are magnified more than horizontal edges within the middle row for even lines, 
 whereby vertical color interpolation and horizontal edge detection is performed for odd lines, but horizontal color interpolation and vertical edge detection is performed for even lines. 
 
     
     
       2. The alternating-line color interpolator and edge detector of  claim 1  wherein diagonal edges that pass through the center location are detected by the edge detector for both odd and even lines. 
     
     
       3. The alternating-line color interpolator and edge detector of  claim 1  wherein the mono-color pixels from the image sensor are arranged in a Bayer pattern. 
     
     
       4. The alternating-line color interpolator and edge detector of  claim 3  wherein the 2-line buffer has a storage capacity sufficient for two full lines of pixels but insufficient for three full lines of the pixels; and
 wherein the 2-line buffer stores less than one percent of the whole image of pixels. 
 
     
     
       5. The alternating-line color interpolator and edge detector of  claim 4  wherein the center location is in the middle row and in the middle column. 
     
     
       6. The alternating-line color interpolator and edge detector of  claim 5  wherein the green color interpolator averages one or two green pixels for each center location, while the red and blue color interpolators each average one, two, or four red or blue pixels for each center location,
 whereby no more than two green pixels are averaged but up to four red or blue pixels are averaged. 
 
     
     
       7. The alternating-line color interpolator and edge detector of  claim 5  wherein the absolute value of the filter coefficient for the center location is at least four times larger than a value of any other of the filter coefficients. 
     
     
       8. The alternating-line color interpolator and edge detector of  claim 3  further comprising:
 an edge enhancer, receiving the edge value from the edge detector, the edge enhancer altering the interpolated green pixel when an edge absolute value being an absolute value of the edge value exceeds a green threshold, p 1  whereby detected edges are enhanced. 
 
     
     
       9. The alternating-line color interpolator and edge detector of  claim 8  wherein the edge enhancer further comprises:
 a first comparator for comparing the edge absolute value to the green threshold, for activating a green multiplier when the edge absolute value exceeds the green threshold; 
 a second comparator for comparing the edge absolute value to a red/blue threshold, for activating a red/blue multiplier when the edge absolute value exceeds the red/blue threshold, the red/blue multiplier altering the interpolated red pixel and the interpolated blue pixel when the edge absolute value exceeds the red/blue threshold, 
 whereby different edge thresholds are compared for green and for red/blue pixels. 
 
     
     
       10. The alternating-line color interpolator and edge detector of  claim 9  wherein the green multiplier multiplies the edge value with a green scale factor and a green summer adds a product from the green multiplier to the interpolated green pixel for adjustment;
 wherein the red/blue multiplier multiplies the edge value with a red/blue scale factor and a red summer adds a product from the red/blue multiplier to the interpolated red pixel while a blue summer adds the product from the red/blue multiplier to the interpolated blue pixel for adjustment, 
 whereby different scaling factors are used for green and for red/blue. 
 
     
     
       11. A digital-image processor comprising:
 a line buffer, receiving mono-color pixels captured by an image sensor, the line buffer for storing only a fraction of a whole frame of an image, the fraction being less than 1 percent of a number of pixels in the whole frame; 
 a merged pipeline, receiving an array of mono-color pixels from the line buffer, for generating missing color values for a middle pixel, the middle pixel being a mono-color pixel in a middle of the array from the line buffer; 
 wherein for odd lines, primary-color pixels in a middle column of the array are averaged to generate a primary-color pixel in the middle of the array while primary-color pixels in other columns are ignored; 
 wherein for even lines, primary-color pixels in a middle row of the array are averaged to generate a primary-color pixel in the middle of the array while primary-color pixels in other rows are ignored; 
 an edge detector, receiving primary-color pixels in the array of mono-color pixels from the line buffer, for detecting an edge between the middle pixel and a neighboring pixel; and 
 an edge enhancer, activated by the edge detector when the edge is detected, for increasing a magnitude of the middle pixel when the edge is detected; 
 the merged pipeline outputting interpolated color values for the middle pixel, 
 whereby interpolation and edge detection are performed by the merged pipeline using the line buffer. 
 
     
     
       12. The digital-image processor of  claim 11  wherein a primary color is green, the primary color pixels are green pixels, and the mono-color pixels are red, green, or blue color values representing a color of a pixel. 
     
     
       13. The digital-image processor of  claim 12  wherein the merged pipeline further comprises:
 a horizontal interpolator, activated for even lines but not activated for odd lines, receiving the array from the line buffer, for averaging primary-color pixels within a same horizontal line to produce a missing color value of the primary color for the middle pixel; and 
 a vertical interpolator, activated for odd lines but not activated for even lines, receiving the array from the line buffer, for averaging primary-color pixels within a same vertical column to produce a missing color value of the primary color for the middle pixel; 
 whereby primary color pixels are horizontally interpolated for even lines but vertically interpolated for odd lines. 
 
     
     
       14. The digital-image processor of  claim 13  wherein the array received by the merged pipeline from the line buffer shifts by one pixel to the right along a line for each clock cycle, the array containing at 3 columns of mono-color pixels from 3 lines;
 wherein the line buffer stores 2 lines of mono-color pixels. 
 
     
     
       15. The digital-image processor of  claim 14  wherein the edge detector comprises:
 an odd-line filter, activated when the middle pixel is on an odd line, for multiplying each primary-color pixel in the array by a filter coefficient and summing products to generate an edge result; 
 wherein filter coefficients for the middle column have an opposite sign as filter coefficients for other columns of the array; and 
 an even-line filter, activated when the middle pixel is on an even line, for multiplying each primary-color pixel in the array by a filter coefficient and summing products to generate an edge result; 
 wherein filter coefficients for the middle row have an opposite sign as filter coefficients for other rows of the array, 
 whereby filter coefficients have a vertical arrangement for odd lines but a horizontal arrangement for even lines. 
 
     
     
       16. The digital-image processor of  claim 5  wherein the odd-line filter detects more vertical and diagonal edges than horizontal edges, while the even line filter detects more horizontal and diagonal edges than vertical edges,
 whereby the edge detector is directional for alternate lines. 
 
     
     
       17. The digital-image processor of  claim 16  wherein the mono-color pixels captured by the image sensor form a Bayer pattern, the Bayer pattern having alternating red and green mono-color pixels on odd lines, and alternating green and blue mono-color pixels on even lines, wherein the primary color pixels are green pixels. 
     
     
       18. An alternating interpolator comprising:
 line buffer means, coupled to receive pixels in a Bayer pattern, for storing lines of pixels, the line buffer means outputting a 3×3 region of pixels having 3 rows and 3 columns; 
 red interpolate means, receiving red pixels in the 3×3 region from the line buffer means, for averaging the red pixels within the 3×3 region to generate an interpolated red pixel for a center location of the 3×3 region; 
 blue interpolate means, receiving blue pixels in the 3×3 region from the line buffer means, for averaging the blue pixels within the 3×3 region to generate an interpolated blue pixel for a center location of the 3×3 region; 
 green interpolate means, receiving green pixels in the 3×3 region from the line buffer means, for averaging the green pixels within the 3×3 region to generate an interpolated green pixel for a center location of the 3×3 region; 
 green-odd means for averaging green pixels in a middle column of the 3×3 region when the center location is on an odd line, the green-odd means excluding green pixels in a left column to the left of the middle column and in a right column to the right of the middle column for odd lines; 
 green-even means for averaging green pixels in a middle row of the 3×3 region when the center location is on an even line, the green-even means excluding green pixels in an upper row above the middle row and in a lower row below the middle row for even lines; and 
 edge detect means, receiving green pixels in the 3×3 region from the line buffer means, for multiplying each green pixel by a filter coefficient and for summing results to generate an edge value; 
 wherein for odd lines, the filter coefficients for green pixels in a middle column have an opposite sign as filter coefficients for green pixels in the right and left column, 
 whereby edges crossing the middle column are detected but vertical edges within the middle column are less detected for odd lines; 
 wherein for even lines, the filter coefficients for green pixels in a middle row have an opposite sign as filter coefficients for green pixels in the upper and lower row, 
 whereby edges crossing the middle row are detected but horizontal edges within the middle row are less detected for even lines, 
 whereby vertical color interpolation and horizontal edge detection is performed for odd lines, but horizontal color interpolation and vertical edge detection is performed for even lines. 
 
     
     
       19. The alternating interpolator of  claim 18  wherein the green color interpolate means averages one or two green pixels for each center location, while the red and blue color interpolate means each average one, two, or four red or blue pixels for each center location,
 whereby no more than two green pixels are averaged but up to four red or blue pixels are averaged. 
 
     
     
       20. The alternating interpolator of  claim 19  further comprising:
 edge enhance means, receiving the edge value from the edge detect means, for altering the interpolated green pixel when an absolute value of the edge value exceeds a green threshold, 
 whereby detected edges are enhanced. 
 
     
     
       21. An apparatus for processing a digital image formed by a total number of pixels, wherein said digital image includes a plurality of lines, said apparatus comprising:
 a line buffer configured to store pixel values, wherein each pixel value is indicative of color information for a corresponding pixel of said digital image, and wherein said line buffer has a capacity for storing a maximum number of pixel values that is less than said total number of pixels of said digital image;   a red color interpolator for receiving said pixel values from said line buffer;   a blue color interpolator for receiving said pixel values from said line buffer;   a green color interpolator for receiving said pixel values from said line buffer; and   an edge-detection unit;   wherein said red color interpolator, said blue color interpolator, and said green color interpolator are configured to generate additional color information corresponding to one or more of said pixels comprising said digital image;   and wherein said edge-detection unit is configured to output greater edge magnification values for horizontal edges than for vertical edges on one of said plurality of lines of said digital image, and wherein said edge-detection unit is configured to output greater edge magnification values for vertical edges than for horizontal edges on a different one of said plurality of lines of said digital image.   
     
     
       22. An apparatus for processing a digital image comprising a plurality of lines of pixels, said apparatus comprising:
 a line buffer configured to store pixel values for a subset of said plurality of lines of pixels, wherein each pixel value is indicative of color information for a corresponding pixel of said digital image;   a plurality of color interpolators configured to receive said pixel values from said line buffer, wherein said plurality of color interpolators are configured to generate additional color information from said pixels values received from said line buffer; and   an edge enhancement unit, wherein said edge enhancement unit is configured to enhance horizontal edges more than vertical edges for one of said plurality of lines of pixels, wherein said edge enhancement unit is configured to enhance vertical edges more than horizontal edges for a different one of said plurality of lines of pixels, and wherein said edge enhancement unit is configured to enhance edges based on edge magnification values.   
     
     
       23. The apparatus of claim 22, further comprising an edge detection unit configured to detect one or more edges within said digital image and to generate edge magnification values for the one or more edges. 
     
     
       24. The apparatus of claim 23, wherein said at least one of said plurality of color interpolators is configured to alternate between horizontal interpolation and vertical interpolation for successive lines of pixels within said digital image. 
     
     
       25. The apparatus of claim 23, wherein, for said one of said plurality of lines of pixels, said edge detection unit is configured to output edge magnification values for vertical edges that are greater in magnitude than edge magnification values for horizontal edges. 
     
     
       26. The apparatus of claim 25, wherein, for said different one of said plurality of lines of pixels, said edge detection unit is configured to output edge magnification values for horizontal edges that are greater in magnitude than edge magnification values for vertical edges. 
     
     
       27. The apparatus of claim 23, wherein said edge detection unit is configured to detect diagonal edges for said one of said plurality of lines of pixels and said different one of said plurality of lines of pixels. 
     
     
       28. The apparatus of claim 22, wherein at least one of said plurality of color interpolators is configured to use horizontal interpolation but not vertical interpolation for generating said additional color information for one of said plurality of lines of pixels, and wherein said at least one of said plurality of color interpolators is configured to use vertical interpolation but not horizontal interpolation for generating said additional color information for a different one of said plurality of lines of pixels. 
     
     
       29. The apparatus of claim 23, wherein said edge detection unit is configured to generate the edge magnification values for the one or more edges based on edge coefficients having differing polarities. 
     
     
       30. The apparatus of claim 22, wherein for one a plurality of colors corresponding to the plurality of color interpolators, said edge enhancement unit is configured to enhance edges for that color using a scaling value, and wherein said edge enhancement unit is configured to enhance edges for a different one the plurality of colors using a different scaling value. 
     
     
       31. A digital camera, comprising:
 a line buffer configured to store pixel values for a subset of a total number of lines in a digital image, wherein each of said pixel values stored in said line buffer includes red color information, blue color information, or green color information;   a red interpolation unit configured to receive pixel values from said line buffer, wherein said red interpolation unit is configured to generate red color information from said pixel values received from said line buffer;   a blue interpolation unit configured to receive pixel values from said line buffer, wherein said blue interpolation unit is configured to generate blue color information from said pixel values received from said line buffer;   a green interpolation unit configured to receive pixel values from said line buffer, wherein said green interpolation unit is configured to generate green color information from said pixel values received from said line buffer;   an edge detection unit configured to receive pixel values from said line buffer, wherein said edge detection unit is configured to use green color information but not red color information or blue color information in order to generate one or more edge values; and   an edge enhancement unit configured to receive color information from said red interpolation unit, said blue interpolation unit, and said green interpolation unit, and an edge value from said edge detection unit.   
     
     
       32. The digital camera of claim 31, wherein said edge enhancement unit is configured to compare said edge value to a red-blue threshold value, wherein if said edge value exceeds said red-blue threshold value, said edge enhancement unit is configured to output enhanced red and blue pixel values based at least in part upon a red-blue scaling factor. 
     
     
       33. The digital camera of claim 32, wherein said edge enhancement unit is configured to compare said edge value to a green threshold value, wherein if said edge value exceeds said green threshold value, said edge enhancement unit is configured to output one or more enhanced green pixel values based at least in part upon a green scaling factor. 
     
     
       34. The digital camera of claim 33, wherein said red-blue scaling factor is different from said green scaling factor. 
     
     
       35. The digital camera of claim 31, wherein said green interpolation unit and said edge detection unit are configured to operate concurrently on a first set of pixel values received from said line buffer.