Bale detection and classification using stereo cameras

Abstract

An apparatus comprises a sensor (102) comprising a left camera (102a) and a right camera (102b). A processor (104) is coupled to the sensor. The processor is configured to produce an image and disparity data for the image, and search for a vertical object (122a, 122b) within the image using the disparity data. The processor is also configured to determine whether the vertical object is a bale of material using the image, and compute an orientation of the bale relative to the sensor using the disparity data. The sensor and processor can be mounted for use on an autonomous bale mover comprising an integral power system, a ground-drive system, a bale loading system, and a bale carrying system.

1 - 95 . (canceled) 
     
     
         96 . A method, comprising:
 scanning a region of land using a sensor comprising stereo cameras;   producing, by the sensor, an image and disparity data for the image;   searching for a vertical object within the image using the disparity data;   determining that the vertical object is a bale of material using the image; and   computing an orientation of the bale relative to the sensor using the disparity data.   
     
     
         97 . The method of  claim 96 , further comprising:
 producing modified disparity data by removing disparity data corresponding to the ground in the image; and   searching for the vertical object within the region using the modified disparity data.   
     
     
         98 . The method of  claim 96 , wherein:
 searching for the vertical object comprises scanning the image using a detection window having a predetermined size in terms of pixels; and   the predetermined size of the detection window corresponds to a size of the bale at a given distance separating the vertical object from the sensor.   
     
     
         99 . The method of  claim 96 , wherein determining that the vertical object is the bale comprises:
 extracting features of the vertical object;   classifying the vertical object using the extracted features and a plurality of classifiers; and   determining that the vertical object is the bale in response to each of the plurality of classifiers successfully classifying the vertical object as the bale.   
     
     
         100 . The method of  claim 96 , wherein determining that the vertical object is the bale comprises:
 classifying, by a first classifier, the vertical object using first features of the object; and   if the first classifier indicates the vertical object is likely the bale, classifying, by a second classifier, the vertical object using second features of the object.   
     
     
         101 . The method of  claim 100 , wherein:
 the first classifier comprises a first support vector machine;   the first features are Haar features;   the second classifier comprises a second support vector machine; and   the second features are HOG (Histogram of Oriented Gradients) features.   
     
     
         102 . The method of  claim 96 , wherein computing bale orientation comprises computing position of the bale relative to the sensor using the disparity data. 
     
     
         103 . The method of  claim 96 , wherein computing bale orientation comprises:
 computing three-dimensional points (X, Y, Z) for the bale within the image using the disparity data;   projecting X and Z coordinates of the three-dimensional points to a two-dimensional (X-Z) plane corresponding to a top-down view of the bale;   determining a face of the bale and a side of the bale using the two-dimensional plane; and   computing the orientation of the bale relative to the sensor using the face of the bale.   
     
     
         104 . The method of  claim 103 , wherein determining the face and side of the bale comprises:
 generating a first best fit line through points in the X-plane;   generating a second best fit line through points in the Z-plane; and   determining which of the first and second best fit lines represents the face of the bale.   
     
     
         105 . The method of  claim 104 , wherein:
 determining which of the first and second best fit lines represents the face of the bale comprises determining disparity data variation for the first and second best fit lines; and   the best fit line with the smallest variation corresponds to the face of the bale.   
     
     
         106 . The method of  claim 96 , further comprising:
 storing orientation and a position of the bale by a world model; and   updating the orientation and position of the bale in the world model in response to subsequent imaging of the bale by the sensor.   
     
     
         107 . The method of  claim 96 , further comprising:
 receiving current orientation and current position of the bale by a world model;   determining variability of the current orientation and current position relative to orientation and position data previously stored in the world model for the bale; and   updating the orientation and position of the bale in the world model to include the current orientation and current position if the variability does not exceed a threshold.   
     
     
         108 . An apparatus, comprising:
 a sensor comprising a left camera and a right camera; and   a processor coupled to the sensor and configured to:
 produce an image and disparity data for the image; 
 search for a vertical object within the image using the disparity data; 
 determine that the vertical object is a bale of material using the image; and 
 compute an orientation of the bale relative to the sensor using the disparity data. 
   
     
     
         109 . The apparatus of  claim 108 , wherein the processor is configured to:
 produce modified disparity data by removing disparity data corresponding to the ground in the image; and   search for the vertical object within the region using the modified disparity data.   
     
     
         110 . The apparatus of  claim 108 , wherein:
 the processor is configured to search for the vertical object by scanning the image using a detection window having a predetermined size in terms of pixels; and   the predetermined size of the detection window corresponds to a size of the bale at a given distance separating the vertical object from the sensor.   
     
     
         111 . The apparatus of  claim 108 , wherein the processor is configured to determine that the vertical object is the bale by:
 extracting features of the vertical object;   classifying the vertical object using the extracted features and a plurality of classifiers; and   determining that the vertical object is the bale in response to each of the plurality of classifiers successfully classifying the vertical object as the bale.   
     
     
         112 . The apparatus of  claim 108 , wherein the processor is configured to determine that the vertical object is the bale by:
 classifying, by a first classifier, the vertical object using first features of the object; and   if the first classifier indicates the vertical object is likely the bale, classifying, by a second classifier, the vertical object using second features of the object.   
     
     
         113 . The apparatus of  claim 112 , wherein:
 the first classifier comprises a first support vector machine;   the first features are Haar features;   the second classifier comprises a second support vector machine; and   the second features are HOG (Histogram of Oriented Gradients) features.   
     
     
         114 . The apparatus of  claim 108 , wherein the processor is configured to compute bale orientation by computing position of the bale relative to the sensor using the disparity data. 
     
     
         115 . The apparatus of  claim 108 , wherein the processor is configured to compute bale orientation by:
 computing three-dimensional points (X, Y, Z) for the bale within the image using the disparity data;   projecting X and Z coordinates of the three-dimensional points to a two-dimensional (X-Z) plane corresponding to a top-down view of the bale;   determining a face of the bale and a side of the bale using the two-dimensional plane; and   computing the orientation of the bale relative to the sensor using the face of the bale.   
     
     
         116 . The apparatus of  claim 115 , wherein the processor is configured to determine the face and side of the bale by:
 generating a first best fit line through points in the X-plane;   generating a second best fit line through points in the Z-plane; and   determining which of the first and second best fit lines represents the face of the bale.   
     
     
         117 . The apparatus of  claim 116 , wherein the processor is configured to:
 determine which of the first and second best fit lines represents the face of the bale by determining disparity data variation for the first and second best fit lines; and   the best fit line with the smallest variation corresponds to the face of the bale.   
     
     
         118 . The apparatus of  claim 108 , wherein the processor is configured to:
 store orientation and a position of the bale by a world model; and   update the orientation and position of the bale in the world model in response to subsequent imaging of the bale by the sensor.   
     
     
         119 . The apparatus of  claim 108 , wherein the processor is configured to:
 receive current orientation and current position of the bale by a world model;   determine variability of the current orientation and current position relative to orientation and position data previously stored in the world model for the bale; and   update the orientation and position of the bale in the world model to include the current orientation and current position if the variability does not exceed a threshold.   
     
     
         120 . The apparatus of  claim 108 , further comprising an autonomous bale mover, wherein the sensor and processor are components of the bale mover.