Object Recognition Using Textons and Shape Filters

Abstract

Given an image of structured and/or unstructured objects, semantically meaningful areas are automatically partitioned from the image, each area labeled with a specific object class. Shape filters are used to enable capturing of some or all of the shape, texture, and/or appearance context information. A shape filter comprises one or more regions of arbitrary shape, size, and/or position within a bounding area of an image, paired with a specified texton. A texton comprises information describing the texture of a patch of surface of an object. In a training process a sub-set of possible shape filters is selected and incorporated into a conditional random field model of object classes. The conditional random field model is then used for object detection and recognition.

1 . A computer-implemented method comprising:
 performed by one or more processors executing computer-readable instructions,
 receiving a plurality of training images of objects; 
 receiving an object label map for each training image, each object label map comprising a label for each image element specifying one of a plurality of object classes; 
 accessing a dictionary of textons, each texton comprising information describing the texture of a patch of surface of an object; 
 forming a texton map for each training image based at least in part on the dictionary of textons, each texton map comprising a label indicating a texton for each image element; 
 forming a shape filter by pairing a bounding area of each training image with a specified texton; 
 for each texton map computing a plurality of feature responses by applying a different shape filter for each feature response; 
 selecting a sub-set of the shape filters used in computing the feature responses by forming a multi-class classifier to classify image elements into the object classes based at least in part on at least one of the feature responses; and 
 forming an object detection and recognition system based at least in part on the selected shape filters. 
   
     
     
         2 . A computer-implemented method as claimed in  claim 1 , wherein each shape filter comprises a bounding area defining an area of an image within which the shape filter is applied, the bounding area being movable within the image. 
     
     
         3 . A computer-implemented method as claimed in  claim 1 , wherein each shape filter comprises a bounding area defining an area of an image within which the shape filter is applied and a plurality of substantially randomly sized and positioned rectangular regions within the bounding area. 
     
     
         4 . A computer-implemented method as claimed in  claim 1 , wherein accessing the dictionary of textons comprises forming the dictionary of textons based at least in part on the training images. 
     
     
         5 . A computer-implemented method as claimed in  claim 1 , wherein the multi-class classifier is formed based at least in part on a joint boosting process. 
     
     
         6 . A computer-implemented method as claimed in  claim 5 , wherein the joint boosting process comprises iteratively building the multi-class classifier as a sum of decision stumps comprising thresholds applied to the feature responses, each decision stump being shared between a plurality of object classes. 
     
     
         7 . A computer-implemented method as claimed in  claim 1 , wherein forming the object detection and recognition system comprises forming a conditional random field model of object classes, the model comprising definitions of a conditional probability of object class labels given an image based at least in part on a plurality of potentials comprising shape-texture potentials based at least in part on the shape filter, texton pairs. 
     
     
         8 . A computer-implemented method as claimed in  claim 7 , further comprising learning parameters for the conditional random field model by dividing the conditional random field model into pieces and training each piece independently based at least in part on a training method incorporating fixed powers. 
     
     
         9 . A computer-implemented method as claimed in  claim 7 , wherein the conditional random field model is formed based at least in part on color potentials arranged to represent a color distribution of an instance of an object class in a particular image. 
     
     
         10 . A computer-implemented method as claimed in  claim 7 , wherein the conditional random field model is formed based at least in part on a location potential. 
     
     
         11 . A computer-implemented method as claimed in  claim 7 , wherein the conditional random field model is formed based at least in part on an edge potential. 
     
     
         12 . A computer-implemented method as claimed in  claim 7 , further comprising:
 determining an overall object labeling for a previously unseen image based at least in part on the conditional random field model; and   inferring an object label map from the determined overall object labeling based at least in part on an inference process.   
     
     
         13 . A computer-implemented method comprising:
 performed by one or more processors executing computer-readable instructions,
 receiving a plurality of training images of objects; 
 receiving an object label map for each training image, each object label map comprising a label for each image element specifying one of a plurality of object classes; 
 accessing a dictionary of textons, each texton comprising information describing the texture of a patch of surface of an object; 
 forming a texton map for each training image based at least in part on the dictionary of textons, each texton map comprising a label indicating a texton for each image element; 
 forming a shape filter by pairing a bounding area of each training image with a specified texton; 
 for each texton map computing a plurality of feature responses by applying a different shape filter for each feature response; 
 selecting a sub-set of the shape filters used in computing the feature responses by forming a multi-class classifier to classify image elements into the object classes based at least in part on at least one of the feature responses; and 
 forming an object label map for a previously unseen image based at least in part on the selected shape filters. 
   
     
     
         14 . A computer-implemented method as claimed in  claim 13 , wherein forming the object label map for the previously unseen image comprises forming a conditional random field model comprising shape-texture potentials, edge potentials, color potentials, or location potentials. 
     
     
         15 . A computer-implemented method as claimed in  claim 14 , further comprising determining parameters for the shape-texture potentials based at least in part on a joint boosting process with a substantially random selection of shape filters. 
     
     
         16 . A computer-implemented method as claimed in  claim 14 , further comprising determining parameters for the color potentials based at least in part on an iterative conditional mode method. 
     
     
         17 . One or more computer-readable storage media storing computer-executable instructions that, when executed on a processor, configure the processor to perform acts comprising:
 receiving a plurality of training images of objects;   receiving an object label map for each training image, each object label map comprising a label for each image element specifying one of a plurality of object classes;   accessing a dictionary of textons, each texton comprising information describing the texture of a patch of surface of an object;   forming a texton map for each training image using the dictionary of textons, each texton map comprising a label indicating a texton for each image element;   forming a shape filter by pairing a bounding area of each training image with a specified texon;   for each texton map computing a plurality of feature responses by applying a different shape filter for each feature response;   selecting a sub-set of the shape filters used in computing the feature responses by forming a multi-class classifier to classify image elements into the object classes based at least in part on at least one of the feature responses; and   forming an object label map based at least in part on the selected shape filters.   
     
     
         18 . The one or more computer-readable storage media of  claim 17 , further comprising applying the shape filters such that each shape filter comprises a bounding area defining an area of an image within which the shape filter is applied, the bounding area being movable within the image. 
     
     
         19 . The one or more computer-readable storage media of  claim 17 , further comprising applying the shape filters such that each shape filter comprises a bounding area having an area of about ½ the image area. 
     
     
         20 . The one or more computer-readable storage media of  claim 17 , further comprising applying the shape filters such that each shape filter comprises a bounding area defining an area of an image within which the shape filter is applied and a plurality of substantially randomly sized and positioned rectangular regions within the bounding area.