Real-time agricultural object detection and display

Abstract

A computer-implemented method includes performing, using a processor onboard a vehicle, a machine learning (ML) processing on sensor input from sensors onboard an agricultural vehicle, identifying, according to a rule, a subset of data resulting from the ML processing and generating and displaying, in real-time, the subset of data to a user interface, thereby enabling a user interaction with the subset of data.

1 . A processor-implemented method, comprising:
 performing, using a processor onboard a vehicle, a machine learning (ML) processing on sensor input from sensors onboard the vehicle;   identifying, according to a rule, a subset of data resulting from the ML processing; and   generating and displaying, in real-time, the subset of data to a user interface, thereby enabling a user interaction with the subset of data.   
     
     
         2 . The method of  claim 1 , wherein the vehicle is an agricultural vehicle operating in an agricultural environment, and wherein the user interface is of a display device mounted on the vehicle or of a portable electronic device controlled by a field operator in the agricultural environment. 
     
     
         3 . The method of  claim 1 , wherein the rule specifies that the subset of data resulting from the ML processing comprises portions of images obtained from the sensor input that, wherein the portions of images include one or more target objects subject to a treatment by a treatment mechanism under control of the processor. 
     
     
         6 . The method of  claim 1 , wherein the rule defines one or more of: the subset of data to include portions of images obtained from the sensor input at a predetermined time interval; the subset of data to include portions of images obtained from the sensor input after a predetermined physical movement of the vehicle; or the subset of data to include portions of images according to a detection criterion upon the ML processing. 
     
     
         7 . The method of  claim 1 , wherein the vehicle is an agricultural vehicle operating in an agricultural environment, and wherein the user interface is displayed on a device that is at a different geographic location than the agricultural environment. 
     
     
         8 . The method of  claim 1 , wherein a treatment module configured to perform a treatment action is disposed on the vehicle and wherein the rule defines the subset as a portion of the sensor input that is expected to include the treatment action on a target object. 
     
     
         9 . The method of  claim 8 , wherein, upon detecting absence of the treatment action in the portion of the sensor input that is expected to include the treatment action on the target object, information is sent to the user interface to enable a human intervention. 
     
     
         10 . A processor-implemented method, comprising:
 performing, using a processor onboard a vehicle, a Machine Learning (ML) processing on sensor input from sensors onboard the vehicle;   identifying, according to a rule, a subset of data resulting from the ML processing; and   providing the subset of data to a user interface.   
     
     
         11 . The method of  claim 10 , wherein the sensor input comprises image data, wherein the ML processing comprises annotating patches of the image data using annotations associated with each patch, wherein each annotation includes a confidence number associated with a confidence level with which the ML processing has identified an object of interest in a corresponding patch of image data. 
     
     
         12 . The method of  claim 11 , wherein the annotation comprises one or more of: adding bounding boxes to objects detected in the image data; adding a syntax element or a pixel to objects detected in the images; or a yes/no selection or a selection from a pre-determined set of selection. 
     
     
         13 . The method of  claim 10 , further including:
 receiving user feedback on the user interface for the subset of data;   generating, based on the user feedback, a training set for further training of an ML model used by the ML processing; and   training the ML model using the training set for future use.   
     
     
         14 . The method of  claim 13 , wherein the training set is generated by:
 determining that a particular confidence number associated with a portion of a particular image is below a threshold;   presenting the portion of the particular image on the user interface of a user device that is at a different geographic location than a location of the vehicle;   receiving an input on the user interface, wherein the input indicates a mode of further processing the particular image; and   selectively including, based on the input on the user interface, the particular image in the training set.   
     
     
         15 . The method of  claim 13 , wherein the training set is generated by:
 determining that a particular confidence number associated with a portion of a particular image is below a threshold;   presenting a real-world location indication for the portion of the particular image on the user interface;   receiving an input on the user interface, wherein the input indicates a mode of further processing the particular image; and   selectively including, based on the input on the user interface, the particular image in the training set.   
     
     
         16 . The method of  claim 10 , wherein the training set is generated as the subset of the image data using active learning in which the subset of images is generated using an exclusion criterion that excludes images similar to images that were previously user-verified, or an inclusion criterion that includes images having an aggregate low confidence level. 
     
     
         17 . The method of  claim 13 , wherein the subset of data includes non-sequential frames of the sensor input and the training set includes intermediate frames between the non-sequential frames, wherein the intermediate frames are used as the training set by propagating user feedback received for the non-sequential frames. 
     
     
         18 . The method of  claim 1 , wherein the vehicle is operating in an agricultural environment and wherein the sensors comprise one or more of a depth sensor, a light detection and ranging (LiDAR) sensor, or an infrared camera. 
     
     
         19 . The method of  claim 1 , wherein the vehicle is operating in an agricultural environment, and wherein the user interface is located on a user device in the agricultural environment. 
     
     
         20 . A computer-implemented method of sensor input processing, comprising:
 performing, using a processor onboard a vehicle, a machine learning (ML) processing on sensor input from sensors onboard the vehicle;   identifying, according to a rule, a subset of data resulting from the ML processing; and   generating the subset of data for modifying the ML processing for a subsequent use.