Stereo image processing using contours

Abstract

A computer-implemented stereo image processing method which uses contours is described. In an embodiment, contours are extracted from two silhouette images captured at substantially the same time by a stereo camera of at least part of an object in a scene. Stereo correspondences between contour points on corresponding scanlines in the two contour images (one corresponding to each silhouette image in the stereo pair) are calculated on the basis of contour point comparison metrics, such as the compatibility of the normal of the contours and/or a distance along the scanline between the point and a centroid of the contour. A corresponding system is also described.

1 . A computer-implemented stereo image processing method comprising, at a processor:
 performing a contour extraction process on first and second silhouette images, captured by a stereo camera, of at least part of an object in a scene, to produce respective first and second pluralities of contour points, each plurality of contour points defining a contour (K, K′) of the at least part of the object;   calculating stereo correspondences between contour points on corresponding scanlines (S) in the first and second pluralities of contour points on the basis of contour point comparison metrics, the contour point comparison metrics comprising at least a centroid separation (distC), comprising a distance along a scanline (S) between each contour point (P, Q) and a centroid of a contour (K, K′) on which the contour point lies.   
     
     
         2 . The method of  claim 1 , comprising receiving a stream of first and second input images at a frame rate, and calculating the stereo correspondences at least at the frame rate so that a 3D contour is output in real time. 
     
     
         3 . The method of  claim 1 , wherein performing the contour extraction process comprises computing a convex hull of the at least part of the object to extract the pluralities of contour points defining the contour (K, K′). 
     
     
         4 . The method of  claim 1 , comprising storing at least one of:
 each set of contour points in an ordered list during computation; and   contour points for each scanline (S) in lists (P, Q).   
     
     
         5 . The method of  claim 1 , wherein calculating stereo correspondences further comprises, for each of a number of corresponding scanlines (S):
 identifying a first set of contour points (P) which lie in the scanline in the first plurality of contour points and a second set of contour points (Q) which lie in the scanline in the second plurality of contour points;   obtaining one or more comparison metrics (n, distC) for each contour point;   comparing the one or more comparison metrics of each of the first set of contour points with the one or more comparison metrics of each of the second set of contour points to produce a cost matrix; and   computing a minimum-cost path through the cost matrix to calculate the stereo correspondences between the first set of contour points and the second set of contour points.   
     
     
         6 . The method of  claim 5 , wherein comparing comparison metrics comprises calculating, for each pair of contour points in the cost matrix, a magnitude of the difference between the centroid separation for one of the pair of contour points and the centroid separation for the other of the pair of contour points. 
     
     
         7 . The method of  claim 1 , comprising imposing one or more constraints on the calculating of stereo correspondences, the one or more constraints comprising at least a box constraint requiring that a dimension of a first bounding box of a first contour (K) defined by the first plurality of contour points must not exceed by more than a predetermined amount a corresponding dimension of a second bounding box of a second contour (K′) defined by the second plurality of contour points. 
     
     
         8 . The method of  claim 1 , comprising computing depth using the calculated correspondences between the first plurality of contour points (P) and the second plurality of contour points (Q) to obtain a depth map. 
     
     
         9 . The method of  claim 8 , further comprising one or more of:
 invalidating contour points (P, Q) which lie in separate, adjacent scanlines (S) along a contour in the depth map and whose depth differs by more than a predetermined amount;   smoothing depth values along a contour in the depth map;   assigning depth values to occluded contour points, invalidated contour points, or both; and   invalidating contour points (P, Q) having a normal vector (n) defining an angle with respect to the scanline (S) that is more than a predetermined amount.   
     
     
         10 . A stereo image processing system comprising:
 a contour extraction module configured to extract contours from each of first and second silhouette images of at least part of an object in a scene, captured by a stereo camera, to produce respective first and second pluralities of contour points, each plurality of contour points defining a contour (K, K′) of the at least part of the object;   a stereo computation module configured to calculate stereo correspondences between contour points on corresponding scanlines in the first and second pluralities of contour points on the basis of contour point comparison metrics, the contour point comparison metrics comprising at least a centroid separation (distC), comprising a distance along a scanline (S) between each contour point (P, Q) and a centroid of a contour (K, K′) on which the contour point lies.   
     
     
         11 . The stereo image processing system of  claim 10 , wherein the stereo computation module is further configured to receive a stream of first and second input images at a frame rate, and calculate the stereo correspondences at least at the frame rate so that a 3D contour is output in real time. 
     
     
         12 . The stereo image processing system of  claim 10 , wherein the contour extraction module is further configured to compute a convex hull of the at least part of the object to extract the pluralities of contour points defining the contour (K, K′). 
     
     
         13 . The stereo image processing system of  claim 10 , comprising a storage module configured to store at least one of:
 each set of contour points in an ordered list during computation; and   contour points for each scanline (S) in lists (P, Q).   
     
     
         14 . The stereo image processing system of  claim 10 , wherein the stereo computation module is further configured to, for each of a number of corresponding scanlines (S):
 identify a first set of contour points (P) which lie in the scanline in the first plurality of contour points and a second set of contour points (Q) which lie in the scanline in the second plurality of contour points;   obtain one or more comparison metrics (n, distC) for each contour point;   compare the one or more comparison metrics of each of the first set of contour points with the one or more comparison metrics of each of the second set of contour points to produce a cost matrix; and   compute a minimum-cost path through the cost matrix to calculate the stereo correspondences between the first set of contour points and the second set of contour points.   
     
     
         15 . The stereo image processing system of  claim 14 , wherein the stereo computation module is further configured to compare comparison metrics at least partly by calculating, for each pair of contour points in the cost matrix, a magnitude of the difference between the centroid separation for one of the pair of contour points and the centroid separation for the other of the pair of contour points. 
     
     
         16 . The stereo image processing system of  claim 10 , comprising a constraint module configured to impose one or more constraints on the calculating of stereo correspondences, the one or more constraints comprising at least a box constraint requiring that a dimension of a first bounding box of a first contour (K) defined by the first plurality of contour points must not exceed by more than a predetermined amount a corresponding dimension of a second bounding box of a second contour (K′) defined by the second plurality of contour points. 
     
     
         17 . The stereo image processing system of  claim 10 , comprising a depth calculation module configured to compute depth using the calculated correspondences between the first plurality of contour points (P) and the second plurality of contour points (Q) to obtain a depth map. 
     
     
         18 . The stereo image processing system of  claim 17 , wherein the depth calculation module is further configured to one or more of:
 invalidate contour points (P, Q) which lie in separate, adjacent scanlines (S) along a contour in the depth map and whose depth differs by more than a predetermined amount;   smooth depth values along a contour in the depth map;   assign depth values to occluded contour points, invalidated contour points, or both; and   invalidate contour points (P, Q) having a normal vector (n) defining an angle with respect to the scanline (S) that is more than a predetermined amount.