Systems and methods for a tailored neural network detector

Abstract

Various embodiments described herein provide for a neural network tailored, based on user-provided input data, to detect user-specified objects in image data. An architecture of an embodiment may use unlabeled data from the user, such as a set of images from a video camera stream, while parameters of a tailored neural network (CNN) are trained or adapted.

1 . (canceled) 
     
     
         2 . A method comprising:
 processing, by one or more hardware processors, raw image data using a first trained neural network to produce a first initial set of region of interest (ROI) pairs, each ROI pair comprising a detected ROI for the raw image data and a detected region label classifying the detected ROI;   processing, by the one or more hardware processors, the raw image data using a second trained neural network to produce a second initial set of ROI pairs;   generating, by the one or more hardware processors, a first intermediate set of ROI pairs by combining the first initial set of ROI pairs and the second initial set of ROI pairs;   evaluating, by the one or more hardware processors, the first intermediate set of ROI pairs using a set of expert classifiers to produce a set of confidence levels for the first intermediate set of ROI pairs;   identifying, by the one or more hardware processors, first and second subsets of ROI pairs, in the first intermediate set of ROI pairs, based on the set of confidence levels, each ROI pair in the first subset of ROI pairs having a confidence level that does not satisfy a first reference confidence level criterion, and each ROI pair in the second subset of ROI pairs having a confidence level that satisfies the first reference confidence level criterion;   sending, by the one or more hardware processors, the first subset of ROI pairs to a labeling system that uses a human individual to confirm or modify a particular detected region label of a particular ROI pair in the first subset of ROI pairs;   receiving, by the one or more hardware processors, a set of human-confirmed ROI pairs received from the labeling system; and   generating, by the one or more hardware processors, a second intermediate set of ROI pairs based on the set of human-confirmed ROI pairs.   
     
     
         3 . The method of  claim 2 , wherein the labeling system is a crowd-sourced annotation system. 
     
     
         4 . The method of  claim 2 , wherein the raw image data comprises a plurality of raw images from at least one of a video data stream or a database. 
     
     
         5 . The method of  claim 2 , wherein the raw image data comprises a plurality of raw images from a camera fixed at a location in a physical environment and having an angle of view of the physical environment, the method comprising:
 causing, by the one or more hardware processors, the second trained neural network to train based on the second intermediate set of ROI pairs, the second trained neural network being trained to process images generated by the camera.   
     
     
         6 . The method of  claim 2 , wherein the identifying of the first and second subsets of ROI pairs, in the first intermediate set of ROI pairs, based on the set of confidence levels comprises for each particular ROI pair in the first intermediate set of ROI pairs:
 determining whether a particular confidence level, in the set of confidence levels, corresponding to the particular ROI satisfies the first reference confidence level criterion; and   including the particular ROI in the first subset of ROI pairs in response to the particular confidence level not satisfying the first reference confidence level criterion; and   including the particular ROI in the second subset of ROI pairs in response to the particular confidence level satisfying the first reference confidence level criterion.   
     
     
         7 . The method of  claim 2 , wherein the combining of the first initial set of ROI pairs and the second initial set of ROI pairs comprises clustering the first initial set of ROI pairs and the second initial set of ROI pairs based at least on one of region size, region position, or region label. 
     
     
         8 . The method of  claim 2 , wherein to produce the second initial set of ROI pairs, the raw image data is processed using the second trained neural network while the second trained neural network is set for a first precision, the method comprising:
 processing, by the one or more hardware processors, the raw image data using the second trained neural network, while the second trained neural network is set for a second precision lower than the first precision, to produce a third initial set of ROI pairs, the generating the second intermediate set of ROI pairs based on the set of human-confirmed ROI pairs comprises combining the third initial set of ROI pairs and the set of human-confirmed ROI pairs.   
     
     
         9 . The method of  claim 8 , wherein the combining of the third initial set of ROI pairs and the set of human-confirmed ROI pairs comprises clustering the third initial set of ROI pairs and the set of human-confirmed ROI pairs based at least on one of region size, region position, or region label. 
     
     
         10 . The method of  claim 8 , wherein the processing of the raw image data using the second trained neural network to produce the third initial set of ROI pairs comprises producing a second set of confidence levels for the third initial set of ROI pairs, the method comprising:
 assigning, by the one or more hardware processors, the second set of confidence levels to the second intermediate set of ROI pairs; and   identifying, by the one or more hardware processors, third and fourth subsets of ROI pairs, in the second intermediate set of ROI pairs, based on the second set of confidence levels, each ROI pair in the third subset of ROI pairs having a confidence level that does not satisfy a second reference confidence level criterion, each ROI pair in the fourth subset of ROI pairs having a confidence level that satisfies the second reference confidence level criterion, and the second reference confidence level criterion assisting in determining which regions of interest are easy for the second trained neural network to label and which regions of interest are hard for the second trained neural network to label.   
     
     
         11 . The method of  claim 10 , comprising:
 storing, by the one or more hardware processors, the third subset of ROI pairs to first training dataset;   storing, by the one or more hardware processors, the fourth subset of ROI pairs to second training dataset; and   causing, by the one or more hardware processors, the second trained neural network to train over the first training dataset and the second training dataset such that the second trained neural network trains over the first training dataset faster than over the second training dataset.   
     
     
         12 . A system comprising:
 a memory storing instructions; and   one or more hardware processors communicatively coupled to the memory and configured by the instructions to perform operations comprising:
 processing raw image data using a first trained neural network to produce a first initial set of region of interest (ROI) pairs, each ROI pair comprising a detected ROI for the raw image data and a detected region label classifying the detected ROI; 
 processing the raw image data using a second trained neural network to produce a second initial set of ROI pairs; 
 generating a first intermediate set of ROI pairs by combining the first initial set of ROI pairs and the second initial set of ROI pairs; 
 evaluating the first intermediate set of ROI pairs using a set of expert classifiers to produce a set of confidence levels for the first intermediate set of ROI pairs; 
 identifying first and second subsets of ROI pairs, in the first intermediate set of ROI pairs, based on the set of confidence levels, each ROI pair in the first subset of ROI pairs having a confidence level that does not satisfy a first reference confidence level criterion, and each ROI pair in the second subset of ROI pairs having a confidence level that satisfies the first reference confidence level criterion; 
 sending the first subset of ROI pairs to a labeling system that uses a human individual to confirm or modify a particular detected region label of a particular ROI pair in the first subset of ROI pairs; 
 receiving a set of human-confirmed ROI pairs received from the labeling system; and 
 generating a second intermediate set of ROI pairs based on the set of human-confirmed ROI pairs. 
   
     
     
         13 . The system of  claim 12 , wherein the labeling system is a crowd-sourced annotation system. 
     
     
         14 . The system of  claim 12 , wherein the raw image data comprises a plurality of raw images from at least one of a video data stream or a database. 
     
     
         15 . The system of  claim 12 , wherein the raw image data comprises a plurality of raw images from a camera fixed at a location in a physical environment and having an angle of view of the physical environment, the operations comprising:
 causing the second trained neural network to train based on the second intermediate set of ROI pairs, the second trained neural network being trained to process images generated by the camera.   
     
     
         16 . The system of  claim 12 , wherein the identifying of the first and second subsets of ROI pairs, in the first intermediate set of ROI pairs, based on the set of confidence levels comprises for each particular ROI pair in the first intermediate set of ROI pairs:
 determining whether a particular confidence level, in the set of confidence levels, corresponding to the particular ROI satisfies the first reference confidence level criterion; and   including the particular ROI in the first subset of ROI pairs in response to the particular confidence level not satisfying the first reference confidence level criterion; and   including the particular ROI in the second subset of ROI pairs in response to the particular confidence level satisfying the first reference confidence level criterion.   
     
     
         17 . The system of  claim 12 , wherein the combining of the first initial set of ROI pairs and the second initial set of ROI pairs comprises clustering the first initial set of ROI pairs and the second initial set of ROI pairs based at least on one of region size, region position, or region label. 
     
     
         18 . The system of  claim 12 , wherein to produce the second initial set of ROI pairs, the raw image data is processed using the second trained neural network while the second trained neural network is set for a first precision, the operations comprising:
 processing the raw image data using the second trained neural network, while the second trained neural network is set for a second precision lower than the first precision, to produce a third initial set of ROI pairs, the generating the second intermediate set of ROI pairs based on the set of human-confirmed ROI pairs comprises combining the third initial set of ROI pairs and the set of human-confirmed ROI pairs.   
     
     
         19 . The system of  claim 18 , wherein the combining of the third initial set of ROI pairs and the set of human-confirmed ROI pairs comprises clustering the third initial set of ROI pairs and the set of human-confirmed ROI pairs based at least on one of region size, region position, or region label. 
     
     
         20 . The system of  claim 18 , wherein the processing of the raw image data using the second trained neural network to produce the third initial set of ROI pairs comprises producing a second set of confidence levels for the third initial set of ROI pairs, the operations comprising:
 assigning, by the one or more hardware processors, the second set of confidence levels to the second intermediate set of ROI pairs; and   identifying, by the one or more hardware processors, third and fourth subsets of ROI pairs, in the second intermediate set of ROI pairs, based on the second set of confidence levels, each ROI pair in the third subset of ROI pairs having a confidence level that does not satisfy a second reference confidence level criterion, each ROI pair in the fourth subset of ROI pairs having a confidence level that satisfies the second reference confidence level criterion, and the second reference confidence level criterion assisting in determining which regions of interest are easy for the second trained neural network to label and which regions of interest are hard for the second trained neural network to label.   
     
     
         21 . A non-transitory computer storage medium comprising instructions that, when executed by a hardware processor of a device, cause the device to perform operations comprising:
 processing raw image data using a first trained neural network to produce a first initial set of region of interest (ROI) pairs, each ROI pair comprising a detected ROI for the raw image data and a detected region label classifying the detected ROI;   processing the raw image data using a second trained neural network to produce a second initial set of ROI pairs;   generating a first intermediate set of ROI pairs by combining the first initial set of ROI pairs and the second initial set of ROI pairs;   evaluating the first intermediate set of ROI pairs using a set of expert classifiers to produce a set of confidence levels for the first intermediate set of ROI pairs;   identifying first and second subsets of ROI pairs, in the first intermediate set of ROI pairs, based on the set of confidence levels, each ROI pair in the first subset of ROI pairs having a confidence level that does not satisfy a first reference confidence level criterion, and each ROI pair in the second subset of ROI pairs having a confidence level that satisfies the first reference confidence level criterion;   sending the first subset of ROI pairs to a labeling system that uses a human individual to confirm or modify a particular detected region label of a particular ROI pair in the first subset of ROI pairs   receiving a set of human-confirmed ROI pairs received from the labeling system; and   generating a second intermediate set of ROI pairs based on the set of human-confirmed ROI pairs.