Systems and Methods for Rapid Development of Object Detector Models

Abstract

A computer vision system configured for detection and recognition of objects in video and still imagery in a live or historical setting uses a teacher-student object detector training approach to yield a merged student model capable of detecting all of the classes of objects any of the teacher models is trained to detect. Further, training is simplified by providing an iterative training process wherein a relatively small number of images is labeled manually as initial training data, after which an iterated model cooperates with a machine-assisted labeling process and an active learning process where detector model accuracy improves with each iteration, yielding improved computational efficiency. Further, synthetic data is generated by which an object of interest can be placed in a variety of setting sufficient to permit training of models. A user interface guides the operator in the construction of a custom model capable of detecting a new object.

We claim: 
     
         1 . A method for developing in one or more processors a student machine learned model for classification and detection of one or more previously specified objects and at least one newly specified object, comprising the steps of
 providing in one or more processors and associated storage a first teacher model comprising a first machine learned model capable of detecting and classifying at least one previously specified object,   providing in one or more processors and associated storage a second teacher model comprising a second machine learned model configured for being trained to detect and classify at least one newly specified object,   providing to the first teacher model and the second teacher model a first training dataset representative of the previously specified objects,   providing to the first teacher model and the second teacher model a second training dataset representative of a newly specified object identified through the use of at least some bounding boxes,   processing, in both the first teacher model and the second teacher model, at least the first training dataset and the second training dataset and generating a first training output and a second training output, respectively,   optimizing the first and second training outputs to generate an optimized training output from the processing step by applying classification algorithms for determining the probability distribution, at an anchor box, of the presence of either the background or any of the objects of interest and applying regression algorithms for determining the bounding box of an object that is detected at the anchor box,   supplying the optimized training output as the student machine learned model configured to classify and detect at least one of the one or more previously specified objects and at least one of the newly specified objects.   
     
     
         2 . The method of  claim 1  wherein at least the second training dataset comprises in part video snippets. 
     
     
         3 . The method of  claim 1  wherein at least the second training dataset comprises in part synthetic data. 
     
     
         4 . The method of  claim 1  wherein the first teacher model and the second teacher model are interoperable. 
     
     
         5 . The method of  claim 4  wherein at least one of the first teacher model and the second teacher model is a single shot multibox detector. 
     
     
         6 . The method of  claim 1  wherein the second training output is provided to an operator for correction and the corrected output is processed in a second iteration of the processing step. 
     
     
         7 . The method of  claim 1  comprising the further step of providing a validation dataset to the first teacher model and the second teacher model. 
     
     
         8 . The method of  claim 1  wherein at least some images are provided to an operator as the result of an uncertainty calculation for distinguishing an object from background. 
     
     
         9 . The method of  claim 8  wherein the uncertainty calculation is based in part on a variable threshold. 
     
     
         10 . The method of  claim 1  wherein a grid of anchor boxes is distributed uniformly throughout an image. 
     
     
         11 . The method of  claim 1  wherein classification is modeled as a softmax function to output confidence of a foreground class or a background class. 
     
     
         12 . The method of  claim 1  wherein regression is modeled as a non-linear multivariate regression function. 
     
     
         13 . The method of  claim 12  wherein the multivariate regression function outputs a four-dimensional vector representing center coordinates, width and height of the bounding box enclosing the object in the image. 
     
     
         14 . The method of  claim 1  wherein the system training output is provided to an operator for correction and the corrected output is processed in a second iteration of the processing step. 
     
     
         15 . The method of  claim 1  in which the second teacher model comprises a plurality of second teacher models, each comprising a second machine learned model capable, following training, of detecting and classifying at least one newly specified object. 
     
     
         16 . The method of  claim 1  further comprising the step of applying at least one of active learning and machine assisted labeling to the output of the second teacher model for correction of missed or mislabeled images. 
     
     
         17 . A system for developing a student machine learned model for classification and detection of one or more previously specified objects and at least one newly specified object comprising
 one or more processors and associated storage coupled to the one or more processors and having stored therein instructions executable by the processors wherein the instructions when executed comprise
 a first machine learned model configured as a first teacher model capable of detecting and classifying one or more previously specified objects identified through the use of at least some anchor bounding boxes, 
 a second machine learned model configured as a second teacher model capable, following training, of detecting and classifying at least one newly specified object, 
 a first training dataset representative of the previously specified objects and a second training dataset representative of a newly specified object identified through the use of at least some bounding boxes, 
   the processors being operable when executing the instructions
 to process, in both the first machine learned model and the second machine learned model, at least the first training dataset and the second training dataset 
 to generate a first training output and a second training output, respectively 
 to optimize the first training output and the second training output in order to generate an optimized training output by applying classification algorithms for determining the probability distribution, at an anchor box, of the presence of either the background or any of the objects of interest and applying regression algorithms for determining the bounding box of an object that is detected at the anchor box, and 
 to supply the optimized training output as the student machine learned model configured to classify and detect at least some of the one or more previously specified objects and at least one of the newly specified objects. 
   
     
     
         18 . The system of  claim 17  wherein at least one of the first and second machine learned models is selected from a group comprising a single shot multibox detector and a low shot learning detector. 
     
     
         19 . The system of  claim 17  in which the second machine learned model comprises a plurality of teacher models, each capable, following training, of detecting and classifying at least one newly specified object. 
     
     
         20 . The system of  claim 17  wherein new unlabeled data is processed in both the first teacher model and the second teacher model. 
     
     
         21 . The system of  claim 17  wherein the optimized training output is provided to an operator for correction and the instructions cause the processor to reiterate execution of the process including the corrected output. 
     
     
         22 . The system of  claim 16  wherein at least one of active learning and machine assisted labeling is applied to the output of the second teacher model for correction of missed or mislabeled images. 
     
     
         23 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
 provide a first teacher model comprising a first machine learned model capable of detecting and classifying one or more previously specified objects identified through the use of at least some anchor bounding boxes,   provide a second teacher model comprising a second machine learned model capable, following training, of detecting and classifying at least one newly specified object,   provide a first training dataset representative of the previously specified objects to the first teacher model and the second teacher model,   provide a second training dataset representative of a newly specified object identified through the use of at least some bounding boxes to the first teacher model and the second teacher model,   process, in both the first teacher model and the second teacher model, at least the first training dataset and the second training dataset and generate a first training output and a second training output, respectively,   optimize the first training output and the second training output by applying classification algorithms for determining the probability distribution, at an anchor box, of the presence of either the background or any of the objects of interest and applying regression algorithms for determining the bounding box of an object that is detected at the anchor box to generate an optimized training output,   supply the optimized training output as the student machine learned model configured to classify and detect at least some of the one or more previously specified objects and at least one of the newly specified objects.   
     
     
         24 . The system of  claim 23  where at least the second training dataset comprises in part video snippets. 
     
     
         25 . The storage media of  claim 23  wherein the second training dataset comprises at least in part video snippets. 
     
     
         26 . The storage media of  claim 23  wherein the second training dataset comprises at least in part synthetic data. 
     
     
         27 . The storage media of  claim 23  wherein classification is modeled as a softmax function to output confidence of a foreground class or a background class. 
     
     
         28 . The storage media of  claim 23  wherein regression is modeled as a non-linear multivariate regression function. 
     
     
         29 . The storage media of  claim 28  wherein the multivariate regression function outputs a four-dimensional vector representing center coordinates, width and height of the bounding box enclosing the object in the image. 
     
     
         30 . The storage media of  claim 23  wherein the software, when executed, applies to the output of the second teacher model at least one of active learning and machine assisted labeling for correction of missed or mislabeled images.