Combining multiple cues in a visual object detection system

Abstract

Systems and methods for detecting visual objects by employing multiple cues include statistically combining information from multiple sources into a saliency map, wherein the information may include color, texture and/or motion in an image where an object is to be detected or background determined. The statistically combined information is thresholded to make decisions with respect to foreground/background pixels.

1 . A method for detecting visual objects by employing multiple cues, comprising the steps of: 
 statistically combining pixel information into a saliency map by weighting multiple cues of a pixel's status using noise estimates as a basis for determining a statistical probability that a pixel is a foreground pixel or background pixel; and    thresholding the statistically combined pixel information to make decisions with respect to the foreground/background pixels.    
   
   
       2 . The method as recited in  claim 1 , wherein the statistically combining includes determining statistical probabilities based on color to determine if a pixel belongs to a background image.  
   
   
       3 . The method as recited in  claim 2 , wherein the determining statistical probabilities includes estimating noise energy in each of red, green and blue color channels for an image as a whole.  
   
   
       4 . The method as recited in  claim 3 , further comprising estimating a gain correction for each pixel in an input image to correct for shadows.  
   
   
       5 . The method as recited in  claim 4 , further comprising forming multiple differences relative to a stable background image to evaluate, relative to the noise energy, estimates for each channel to determine a standard deviation from a mean value.  
   
   
       6 . The method as recited in  claim 1 , wherein the statistically combining includes determining probabilities based on motion to determine if a pixel belongs to a background image.  
   
   
       7 . The method as recited in  claim 6 , further comprising confining motion energy to an interior of an object by taking differences between image frames in a monochrome version of the image.  
   
   
       8 . The method as recited in  claim 1 , wherein the statistically combining includes determining probabilities based on texture to determine if a pixel belongs to a background image.  
   
   
       9 . The method as recited in  claim 8 , further comprising computing multiple difference measures for edges by using a normalized monochrome image.  
   
   
       10 . The method as recited in  claim 9 , wherein the difference measures include at least one of a Sobel horizontal mask convolution, H, a Sobel vertical mask convolution, V, and a center surround difference for neighboring pixels.  
   
   
       11 . The method as recited in  claim 1 , wherein the statistically combining includes combining probabilities for color differences, texture differences and motion for each pixel in an image, and based upon the combined probability, determining if the pixel is background.  
   
   
       12 . The method as recited in  claim 1 , wherein the statistically combining information includes the adjusting the statistical probabilities to permit combining the probabilities.  
   
   
       13 . A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for detecting visual objects by employing multiple cues, as recited in  claim 1 .  
   
   
       14 . A method for detecting visual objects by employing multiple cues, comprising the steps of: 
 for each pixel, determining a probability for each of a plurality of information sources for making a determination as to a status of the pixel using noise estimates as a basis for determining the probability that a pixel is a foreground pixel or background pixel;    statistically combining the probabilities from all of the information sources to form a saliency map to determine whether a pixel belongs to a background image or an object; and    thresholding the statistically combined information to make decisions with respect to foreground/background pixels.    
   
   
       15 . The method as recited in  claim 14 , wherein an information source includes pixel color and the determining probability is based on color to determine if a pixel belongs to a background image.  
   
   
       16 . The method as recited in  claim 15 , wherein the determining probability includes estimating and accounting for noise energy in each of the red, green and blue color channels for an image as a whole.  
   
   
       17 . The method as recited in  claim 16 , further comprising estimating a gain correction for each pixel in an input image to correct for shadows.  
   
   
       18 . The method as recited in  claim 17 , further comprising forming multiple differences relative to a stable background image to evaluate, relative to noise energy, estimates for each channel.  
   
   
       19 . The method as recited in  claim 14 , wherein the determining probability is based on motion to determine if a pixel belongs to a background image.  
   
   
       20 . The method as recited in  claim 19 , further comprising the step of confining motion energy to an interior of an object by taking differences between image frames in a monochrome version of the image.  
   
   
       21 . The method as recited in  claim 14 , wherein the determining a probability is based on texture to determine if a pixel belongs to a background image.  
   
   
       22 . The method as recited in  claim 21 , further comprising computing multiple difference measures for edges by using a normalized monochrome image.  
   
   
       23 . The method as recited in  claim 22 , wherein the difference measures include at least one of a Sobel horizontal mask convolution, H, a Sobel vertical mask convolution, V, and a center surround difference for neighboring pixels.  
   
   
       24 . The method as recited in  claim 14 , wherein the statistically combining includes combining probabilities for color differences, texture differences and motion for each pixel in an image, and based upon the combined probability determining if the pixel is background.  
   
   
       25 . The method as recited in  claim 14 , wherein the statistically combining includes the step of adjusting the probabilities to permit combining the probabilities.  
   
   
       26 . A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for detecting visual objects by employing multiple cues, as recited in  claim 14 .  
   
   
       27 . A system for detecting visual objects by employing multiple cues, comprising: 
 a video source which provides images to be processed;    a probability determination module which determines a probability for a plurality of cues based upon available information to determine if a pixel belongs to an object or a background, wherein the cues include a combination of pixel-by-pixel cues and local neighborhood cues; and    a statistical combiner which combines the probabilities from each of the plurality of cues into a saliency map such that statistically combined information is employed to make decisions with respect to foreground or background pixels.    
   
   
       28 . The system as recited in  claim 27  wherein the cues are related to at least one of color, texture and motion in an image where an object is to be detected.  
   
   
       29 . The system as recited in  claim 27  wherein the probability determination module and the statistical combiner are included in a computer system.  
   
   
       30 . The system as recited in  claim 27  further comprising a noise estimator, which estimates noise, which is employed in deriving probabilities for the cues.  
   
   
       31 . A system for detecting visual objects by employing multiple cues, comprising: 
 a video source which provides images to be processed;    a probability determination module which determines a probability for a plurality of cues based upon available information to determine if a pixel belongs to an object or a background;    a noise estimator, which estimates noise for each cue, wherein the noise estimate is employed in deriving probabilities for the cues; and    a statistical combiner which combines the probabilities from each of the plurality of cues into a saliency map such that statistically combined information is employed to make decisions with respect to foreground or background pixels.    
   
   
       32 . The system as recited in  claim 31 , wherein the cues include a combination of pixel-by-pixel cues and local neighborhood cues.