Merged pipeline for color interpolation and edge enhancement of digital images

Abstract

A digital-camera processor receives mono-color digital pixels from an image sensor. Each mono-color pixel is red, blue, or green. The stream of pixels from the sensor has alternating green and red pixels on odd lines, and blue and green pixels on even lines in a Bayer pattern. Each mono-color pixel is white balanced by multiplying with a gain determined in a previous frame and then stored in a line buffer. A horizontal interpolator receives an array of pixels from the line buffer. The horizontal interpolator generates missing color values by interpolation within horizontal lines in the array. The intermediate results from the horizontal interpolator are stored in a column buffer, and represent one column of pixels from the line buffer. A vertical interpolator generates the final RGB value for the pixel in the middle of the column register by vertical interpolation. The RGB values are converted to YUV. The vertical interpolator also generates green values for pixels above and below the middle pixel. These green values are sent to an edge detector. The edge detector applies a filter to the 3 green values and 6 more green values from the last 2 clock cycles. When an edge is detected, an edge enhancer is activated. The edge enhancer adds a scaled factor to the Y component to sharpen the detected edge. Color enhancement is performed on the U and V components. The line buffer stores only 4 full lines of pixels and no full-frame buffer is needed.

1. A digital-image processor comprising:
 a line buffer, receiving mono-color pixels captured by an image sensor, the line buffer for storing only a fraction of a whole frame of an image, the fraction being less than 5 percent of a number of pixels in the whole frame; 
 a merged pipeline, receiving an array of mono-color pixels from the line buffer, for generating missing color values for a middle pixel, the middle pixel being a mono-color pixel in a middle of the array from the line buffer; 
 the merged pipeline also for generating an upper primary color value for an upper pixel immediately above the middle pixel and a lower primary color value for a lower pixel immediately below the middle pixel; 
 an edge detector, receiving the upper primary color value and the lower primary color value and a primary color value for the middle pixel, the edge detector for detecting an edge between the middle pixel and a neighboring pixel; and 
 an edge enhancer, activated by the edge detector when the edge is detected, for increasing a magnitude of the middle pixel when the edge is detected; 
 the merged pipeline outputting interpolated color values for the middle pixel, 
 whereby interpolation and edge detection are performed by the merged pipeline using the line buffer. 
 
     
     
       2. The digital-image processor of  claim 1  further comprising:
 a color-space converter, coupled to receive the middle pixel including the missing color values generated by the merged pipeline, for converting color values into luminance and chromatic values; 
 wherein the edge enhancer increases the luminance value of the middle pixel when activated. 
 
     
     
       3. The digital-image processor of  claim 2  wherein the color-space converter converts from red, green, blue RGB color values to luminance and chromatic YUV values representing a color of a pixel. 
     
     
       4. The digital-image processor of  claim 3  wherein the edge detector receives green color values, the edge detector performing edge detection using green values, but the edge enhancer sharpening edges by adjusting the luminance value of the middle pixel. 
     
     
       5. The digital-image processor of  claim 1  wherein the merged pipeline further comprises:
 a horizontal interpolator, receiving the array from the line buffer, for averaging mono-color pixels within a same horizontal line to produce intermediate color values; 
 a column register, coupled to receive the intermediate color values from the horizontal interpolator, for storing intermediate color values for a single column of pixels in the array; and 
 a vertical interpolator, receiving the intermediate color values from the column register, for averaging intermediate color values within a same single column of pixels, the vertical interpolator outputting RGB color values for the middle pixel, the vertical interpolator also outputting G values as the upper and lower primary color values to the edge detector, 
 wherein horizontal and vertical interpolation are performed separately. 
 
     
     
       6. The digital-image processor of  claim 5  wherein the array received by the merged pipeline from the line buffer shifts by one pixel to the right along a line for each clock cycle, the array containing at 3 columns of mono-color pixels from at least 5 lines;
 wherein the line buffer stores 4 lines of mono-color pixels. 
 
     
     
       7. The digital-image processor of  claim 6  wherein the column register includes one green register for each of at least 5 lines, the column register also including at least 3 shared registers for a middle line containing the middle pixel and an upper and a lower line immediately above and below the middle line;
 wherein the shared registers alternately store red and blue color values for alternate clock cycles. 
 
     
     
       8. The digital-image processor of  claim 7  wherein the line buffer includes a temporary buffer for storing a portion of a fifth line, the portion being part of the array sent to the merged pipeline. 
     
     
       9. The digital-image processor of  claim 5  wherein the mono-color pixels are not stored in a whole-frame buffer, but only in the line buffer and in pipeline registers, wherein all the pipeline registers together store less than a line of pixels,
 whereby whole-frame buffers are not needed. 
 
     
     
       10. The digital-image processor of  claim 5  wherein the mono-color pixels captured by the image sensor form a Bayer pattern, the Bayer pattern having alternating red and green mono-color pixels on odd lines, and alternating green and blue mono-color pixels on even lines. 
     
     
       11. The digital-image processor of  claim 5  wherein the fraction of the whole frame of the image that is stored by the line buffer is less than one percent of the whole frame,
 whereby the line buffer is 99 percent smaller than a whole-frame buffer. 
 
     
     
       12. The digital-image processor of  claim 5  further comprising:
 a white-balance multiplier, coupled to receive the mono-color pixels from the image sensor, for multiplying red pixels by a red gain, for multiplying green pixels by a green gain, and for multiplying blue pixels by a blue gain, the white-balance multiplier outputting white-balanced mono-color pixels to the line buffer, 
 whereby the mono-color pixels are white-balanced on-the-fly before being written to the line buffer. 
 
     
     
       13. A method for processing digital images comprising:
 receiving a series of mono-color pixels, each pixel representing one of three primary colors; 
 white-balancing the mono-color pixels by multiplying each mono-color pixel by a color-specific gain to produce balanced pixels; 
 writing the balanced pixels to a line buffer containing only a few lines of an image; 
 selecting a range within the line buffer, the range surrounding a target pixel; 
 horizontally interpolating balanced pixels for each line within the range selected from the line buffer, to produce intermediate components; 
 storing the intermediate components and some of the mono-color pixels within one column of the range selected from the line buffer in a column register; 
 vertically interpolating the intermediate components and mono-color pixels from the column register to produce a full-color pixel for the target pixel, and also producing a primary-color component for a pixel above the target pixel and a pixel below the target pixel; 
 detecting an edge at the target pixel by computing a filter value from the primary-color components of the target pixel and of the pixel above and the pixel below; 
 when the filter value exceeds a threshold, enhancing the edge by adjusting the target pixel; and 
 outputting the full-color pixel after adjustment for edge enhancement as the target pixel, 
 whereby the target pixel is interpolated and edge enhanced using outputs from vertical interpolation. 
 
     
     
       14. The method of  claim 13  further comprising:
 converting the full-color pixel to a YUV pixel that includes a Y brightness and U and V color components; 
 wherein enhancing the edge by adjusting the target pixel comprises adjusting the Y brightness but not the U and V color components of the YUV pixel; 
 wherein outputting the full-color pixel after adjustment comprises outputting the YUV pixel after Y adjustment for edge enhancement. 
 
     
     
       15. The method of  claim 14  wherein horizontally interpolating balanced pixels comprises:
 right-shifting a pair of digital values for mono-color pixels to weight the digital values; 
 adding right-shifted digital values for mono-color pixels to the immediate right and left of the target pixel to produce an intermediate component, 
 whereby horizontal interpolation is performed by shifting and adding adjacent mono-color pixels on a line. 
 
     
     
       16. The method of  claim 15  further comprising:
 color enhancing the full-color pixel by adjusting the U and V color components of the YUV pixel but not adjusting the Y brightness; 
 wherein outputting the full-color pixel after adjustment comprises-outputting the YUV pixel after Y adjustment for edge enhancement and U and V adjustment for color enhancement, 
 whereby color enhancement is performed. 
 
     
     
       17. A single-pass interpolator and edge detector comprising:
 line-buffer means, receiving partial pixels with incomplete color information, for storing at least 2 lines of pixels of an image being processed but storing less than one-tenth of the image; 
 horizontal interpolate means, receiving a range of pixels from the line-buffer means, for processing pixels within the range through a horizontal filter, the horizontal filter operating only on pixels in a same line, the horizontal interpolate means outputting intermediate results; 
 column means, coupled to the horizontal interpolate means, for holding the intermediate results from a single column, the intermediate results including color information not present in a partial pixel but interpolated from adjacent pixels in a line; 
 vertical interpolate means, receiving the intermediate results from the column means, for processing the intermediate results to generate full-color information for a target pixel in the range, the full color information including an edge color, and for generating the edge color but not the full-color information for a pair of surrounding pixels adjacent to the target pixel; and 
 edge detect means, receiving the edge color for the target pixel and for the pair of surrounding pixels from the vertical interpolate means, for detecting an edge within the image, 
 whereby full-color information is interpolated from partial pixels while edge detection uses interpolation results without an intermediate full-frame buffer between the vertical interpolate means and the edge detect means. 
 
     
     
       18. The single-pass interpolator and edge detector of  claim 17  further comprising:
 color-space conversion means, receiving the full-color information for the target pixel, for converting the full-color information to brightness and hue information for output; 
 edge enhance means, activated by the edge detect means, for adjusting a brightness of the target pixel when an edge is detected, 
 whereby edges are detected from the edge color, but brightness is adjusted. 
 
     
     
       19. The single-pass interpolator and edge detector of  claim 18  wherein the column means stores intermediate results from a single column of the range of pixels read from the line-buffer means,
 whereby a single column of intermediate results are sent from the horizontal interpolate means to the vertical interpolate means. 
 
     
     
       20. The single-pass interpolator and edge detector of  claim 19  wherein the horizontal interpolate means includes add means for adding color values of adjacent pixels;
 wherein the vertical interpolate means includes shift means for weighting color values and add means for adding weighted color values of adjacent pixels. 
 
     
     
       21. An apparatus, comprising:
 a line buffer configured to receive mono-color pixel values corresponding to pixels in a digital image;   a color interpolation unit configured to receive mono-color pixel values from the line buffer and generate one or more additional color pixel values in a first color format;   a color converter unit configured to receive color pixel values from the color interpolation unit and convert received pixel values from the first color format to a second color format; and   an edge detection unit configured to detect an edge within the digital image;   wherein, for a given pixel in the digital image, the color converter unit and the edge detection unit are configured to perform color conversion and edge detection in parallel using the color pixel values for the given pixel that are generated by the color interpolation unit.   
     
     
       22. The apparatus of claim 21, wherein the color converter unit includes a register configured to store a red value, a green value, and a blue value, and wherein the color converter unit includes an RGB to YUV converter. 
     
     
       23. The apparatus of claim 21, further comprising a processing unit including the color interpolation unit, the color converter unit, and the edge detection unit. 
     
     
       24. The apparatus of claim 23, wherein the processing unit includes an edge enhancement unit. 
     
     
       25. The apparatus of claim 24, wherein the edge enhancement unit is configured to perform a processing operation on one or more pixel values in response to the edge detection unit detecting an edge within the digital image. 
     
     
       26. The apparatus of claim 25, wherein the edge enhancement unit is configured to increase a luminance value of one or more pixel values in response to the edge detection unit detecting an edge within the digital image. 
     
     
       27. The apparatus of claim 23, wherein the line buffer is configured to receive the pixel values from an image sensor, and wherein the line buffer and the processing unit are configured to process each of the pixels in the digital image without storing the pixels in a frame buffer. 
     
     
       28. The apparatus of claim 27, wherein the color interpolation unit includes a horizontal interpolation unit configured to perform interpolation based on pixel values within a same horizontal line of pixels. 
     
     
       29. The apparatus of claim 27, wherein the color interpolation unit includes a vertical interpolation unit configured to perform interpolation based on pixel values within a same column of pixels. 
     
     
       30. The apparatus of claim 28, wherein the color interpolation unit includes a vertical interpolation unit configured to perform interpolation based on pixel values within a same column of pixels. 
     
     
       31. The apparatus of claim 21, wherein each of the received mono-color pixel values is either a red, green, or blue color value, and wherein the edge detection unit is configured to perform edge detection based only on green pixel values. 
     
     
       32. A method, comprising:
 receiving, at a line buffer, mono-color pixel values corresponding to pixels in a digital image;   at a color interpolation unit, generating one or more additional color pixel values corresponding to the mono-color pixel values and outputting intermediate pixel values in a first color format;   a color converter unit converting pixel values in the first color format to pixel values in a second color format; and   an edge detection unit detecting an edge within the digital image;   wherein said converting pixel values and said detecting an edge are performed in parallel.   
     
     
       33. The method of claim 32, further comprising:
 a color enhancer unit performing color enhancement on pixel values in the second color format; and   an edge enhancer unit performing edge enhancement on pixel values in the second color format;   wherein, for a given pixel value, said performing color enhancement and said performing edge enhancement occurs at the same time.   
     
     
       34. The method of claim 33, wherein the second color format is YUV;
 wherein the edge detection unit is configured to detect edges based only on green pixel values; and   wherein the edge enhancer is configured to receive:
 a Y value from the color converter unit; and 
 input from the edge detection unit. 
   
     
     
       35. The method of claim 32, further comprising storing the digital image. 
     
     
       36. The method of claim 32, wherein the color converter unit includes a color enhancer and a register configured to store a red value, a green value, and a blue value. 
     
     
       37. The method of claim 32, wherein the first color format includes red, green, and blue values and the second color format includes luminance and chrominance values. 
     
     
       38. The method of claim 32, wherein the method is performed by a digital camera. 
     
     
       39. An apparatus, comprising:
 first means for receiving mono-color pixel values corresponding to pixels in a digital image;   second means for receiving digital pixel values from the first means and generating additional color pixel values in a first color format;   third means for converting pixel values from the first color format to a second color format; and   fourth means for detecting an edge within the digital image;   wherein, for a given pixel in the digital image, the third means and the fourth means are configured to operate in parallel.   
     
     
       40. The apparatus of claim 39, further comprising
 fifth means for enhancing color of pixel values in the second color format; and   sixth means for enhancing one or more edges within the digital image;   wherein, for a given pixel in the digital image, the fifth means and the sixth means are configured to operate at the same time.   
     
     
       41. The apparatus of claim 39, wherein the first color format is RGB. 
     
     
       42. The apparatus of claim 39, wherein the second color format is YUV. 
     
     
       43. The apparatus of claim 39, wherein the apparatus is a digital camera. 
     
     
       44. A method, comprising:
 generating a frame of digital mono-color pixel values at an image sensor in a digital camera;   processing digital pixel values received from said image sensor to produce a digital image, wherein said processing is performed without using a whole frame buffer;   wherein said processing further includes:
 performing color interpolation on said mono-color pixel values to produce pixel values in a first color format; 
 converting pixel values from the first color format to a second color format; and 
 detecting an edge in the digital image; 
 wherein for a given pixel in the digital image, said converting and said detecting occur in parallel. 
   
     
     
       45. The method of claim 44, wherein said processing includes white balancing pixel values. 
     
     
       46. The method of claim 44, wherein said processing includes storing pixel values in a line buffer. 
     
     
       47. The method of claim 44, wherein the first color format is RGB. 
     
     
       48. The method of claim 44, wherein the second color format is YUV. 
     
     
       49. The method of claim 48, wherein said processing includes:
 enhancing colors of pixel values in the second color format; and   enhancing one or more edges in the digital image;   wherein said enhancing colors and said enhancing one or more edges, for a given pixel in the digital image, occur at the same time.   
     
     
       50. The method of claim 49, wherein said enhancing one or more edges includes altering a luminance value. 
     
     
       51. The method of claim 44, wherein said performing color interpolation includes using vertical and horizontal interpolation to generate additional color values for mono-color pixels. 
     
     
       52. The method of claim 44, wherein said processing includes performing color enhancement. 
     
     
       53. The method of claim 44, wherein said processing includes performing edge enhancement and color enhancement concurrently on a given set of pixel values. 
     
     
       54. An apparatus, comprising:
 a line buffer configured to store digital pixel values received from an image sensor; and   a pipeline coupled to the line buffer;   wherein the pipeline is configured to receive digital pixel values from the line buffer and to generate final pixel values usable to display and/or store a digital image;   wherein the pipeline is configured to:
 for a given pixel in the digital image, perform color conversion and edge detection on the given pixel in parallel; and 
 for a given pixel in the digital image for which an edge is detected, perform color enhancement and edge enhancement in parallel; 
   wherein the apparatus is configured generate the final pixel values without using a whole frame buffer.   
     
     
       55. The apparatus of claim 54, wherein the pipeline includes a color interpolation unit configured to receive a plurality of mono-color pixel values from the line buffer and to generate additional color values for each pixel position. 
     
     
       56. The apparatus of claim 55, wherein the color interpolation unit includes separate horizontal and vertical interpolation units, each of which is configured to use color values from neighboring pixels to generate additional color values for a current pixel. 
     
     
       57. The apparatus of claim 56, wherein the color interpolation unit is configured to use separate interpolation filters for each color value produced by the color interpolation unit. 
     
     
       58. The apparatus of claim 55, wherein the mono-color pixel values are either red, blue, or green, and wherein, for each received mono-color pixel value, the color interpolation unit is configured to generate two additional color values, wherein the two additional color values are red and blue if the received mono-color pixel value is green, wherein the two additional color values are red and green if the received mono-color pixel value is blue, and wherein the two additional color values are blue and green if the received mono-color pixel value is red. 
     
     
       59. The apparatus of claim 55, wherein the color interpolation unit is coupled to an edge detection unit and a color conversion unit, and wherein the color interpolation unit is configured to output pixel color values to the edge detection unit and the color conversion unit, wherein the edge detection unit and the color conversion unit are configured to operate concurrently on a given set of pixels. 
     
     
       60. The apparatus of claim 54, wherein the pipeline includes a color interpolation unit configured to receive a plurality of mono-color pixel values from the line buffer, wherein the mono-color pixel values are either red, blue, or green, and wherein the color interpolation unit is configured to generate additional color values for a received pixel value only if the received pixel value is red or blue. 
     
     
       61. The apparatus of claim 54, wherein the pipeline includes first means for performing color interpolation upon mono-color pixel values received from the line buffer. 
     
     
       62. The apparatus of claim 61, wherein the pipeline includes second means for performing said color conversion on pixel values received from the first means, and wherein the pipeline includes third means for performing said edge detection on pixel values received from the first means. 
     
     
       63. The apparatus of claim 62, wherein the second means and the third means are configured to operate concurrently on a given set of pixel values. 
     
     
       64. The apparatus of claim 54, wherein the pipeline includes a color interpolation unit coupled to a color conversion unit via a pixel register, wherein the pixel register is configured to store a plurality of color values for a given pixel position. 
     
     
       65. The apparatus of claim 54, wherein the pipeline includes a color interpolation unit coupled to a edge detection unit via a pixel storage unit configured to store pixel values from at least a current column. 
     
     
       66. The apparatus of claim 65, wherein the edge detection unit is configured to receive only green color values of a given pixel position. 
     
     
       67. The apparatus of claim 54, wherein the pipeline includes a color conversion unit. 
     
     
       68. The apparatus of claim 67, wherein the color conversion unit is configured to convert RGB color values to YUV values. 
     
     
       69. The apparatus of claim 54, wherein the pipeline includes first means for converting incoming color values in a first color format to outgoing color values in a second color format. 
     
     
       70. The apparatus of claim 69, wherein the first color format includes red, blue, and green color values. 
     
     
       71. The apparatus of claim 69, wherein the second color format includes luminance and chrominance values. 
     
     
       72. The apparatus of claim 70, wherein the second color format includes luminance and chrominance values. 
     
     
       73. The apparatus of claim 54, wherein the pipeline includes an edge detection unit configured to perform edge detection. 
     
     
       74. The apparatus of claim 54, wherein the edge detection unit is configured to perform edge detection using only green color values. 
     
     
       75. The apparatus of claim 54, wherein the pipeline includes first means for performing edge detection on received pixel values. 
     
     
       76. The apparatus of claim 75, wherein the first means is configured to perform edge detection using green color values and not other color values. 
     
     
       77. The apparatus of claim 54, wherein the pipeline includes a color enhancement unit configured to modify color values received as inputs. 
     
     
       78. The apparatus of claim 54, wherein the color enhancement unit is directly coupled to a color conversion unit. 
     
     
       79. The apparatus of claim 78, wherein the color enhancement unit is configured to receive input luminance and chrominance values and is further configured to generate output luminance and chrominance values. 
     
     
       80. The apparatus of claim 54, wherein the pipeline further includes first means for performing color enhancement of received pixel color values. 
     
     
       81. The apparatus of claim 54, wherein the pipeline includes an edge enhancement unit coupled to adjust a luminance value of a pixel upon detection of an edge. 
     
     
       82. The apparatus of claim 54, wherein the pipeline includes first means for adjusting a luminance value of a pixel in response to detection of an edge.