Systems and methods for dataset and model management for multi-modal auto-labeling and active learning

Abstract

Datasets for autonomous driving systems and multi-modal scenes may be automatically labeled using previously trained models as priors to mitigate the limitations of conventional manual data labeling. Properly versioned models, including model weights and knowledge of the dataset on which the model was previously trained, may be used to run an inference operation on unlabeled data, thus automatically labeling the dataset. The newly labeled dataset may then be used to train new models, including sparse data sets, in a semi-supervised or weakly-supervised fashion.

The invention claimed is: 
     
       1. A method for managing a dataset comprising:
 receiving, at a device that is remotely located from an agent, sensor data collected by one or more sensors associated with the agent and a first dataset including an unlabeled object and one or more labeled objects in a set of frames captured via the one or more sensors over a period of time, the dataset and the sensor being received based on a connection bandwidth at the agent being greater than a bandwidth threshold, the one or more labeled objects being labeled by a first trained network at the agent; 
 inferring, at a second trained network at the device, a new label for the unlabeled object based on the one or more labeled object in the first dataset and a second dataset including one or more objects similar to the unlabeled object, the new label being different than a respective label associated each of the one or more labeled objects; 
 labeling the unlabeled object with the new label to generate a third dataset; and 
 training an untrained network using the third dataset to generate a new model. 
 
     
     
       2. The method of  claim 1  wherein the first dataset comprises data from a group of sensors. 
     
     
       3. The method of  claim 2  wherein the group of sensors include two or more of a LiDAR sensor, a RADAR sensor, an RGB camera, monocular camera, and/or stereo camera. 
     
     
       4. The method of  claim 1  wherein the first trained network comprises a versioned model. 
     
     
       5. The method of  claim 4  wherein the versioned model comprises a weighted model. 
     
     
       6. The method of  claim 4  wherein the versioned model comprises an identification of a training dataset. 
     
     
       7. The method of  claim 1  further comprising tracking the new model as auto-labeled. 
     
     
       8. The method of  claim 1  further comprising tracking the first trained network and first dataset. 
     
     
       9. The method of  claim 1  wherein the untrained network is trained in a semi-supervised manner. 
     
     
       10. The method of  claim 1  wherein the first dataset is labeled according to a first ontology. 
     
     
       11. The method of  claim 10  wherein the first dataset is also labeled according to a second ontology. 
     
     
       12. An apparatus for managing a dataset comprising:
 at least one processor; and 
 at least one memory coupled with the at least one processor and storing instructions operable, when executed by the at least one processor, to cause the apparatus to:
 receive, at a device that is remotely located from an agent, sensor data collected by one or more sensors associated with the agent and a first dataset including an unlabeled object and one or more labeled objects in a set of frames captured via the one or more sensors over a period of time, the dataset and the sensor being received based on a connection bandwidth at the agent being greater than a bandwidth threshold, the one or more labeled objects being labeled by a first trained network at the agent; 
 infer, at a second trained network at the device, a new label for the unlabeled object based on the one or more labeled object in the first dataset and a second dataset including one or more objects similar to the unlabeled object, the new label being different than a respective label associated each of the one or more labeled objects; 
 label the unlabeled object with the new label to generate a third dataset; and 
 train an untrained network using the third dataset to generate a new model. 
 
 
     
     
       13. The apparatus of  claim 12  wherein the dataset comprises data from a group of sensors. 
     
     
       14. The apparatus of  claim 13  wherein the group of sensors include two or more of a LiDAR sensor, a RADAR sensor, an RGB camera, monocular camera, and/or stereo camera. 
     
     
       15. The apparatus of  claim 12  wherein the first trained network comprises a versioned model. 
     
     
       16. The apparatus of  claim 15  wherein the versioned model comprises a weighted model. 
     
     
       17. The apparatus of  claim 15  wherein the versioned model comprises an identification of a training dataset. 
     
     
       18. The apparatus of  claim 12  wherein the new model is tracked as auto-labeled. 
     
     
       19. The apparatus of  claim 12  wherein the untrained network is trained in a semi-supervised manner. 
     
     
       20. A non-transitory computer-readable medium for managing a dataset and including instructions that when executed by one or more processors cause the one or more processors to:
 receive, at a device that is remotely located from an agent, sensor data collected by one or more sensors associated with the agent and a first dataset including an unlabeled object and one or more labeled objects in a set of frames captured via the one or more sensors over a period of time, the dataset and the sensor being received based on a connection bandwidth at the agent being greater than a bandwidth threshold, the one or more labeled objects being labeled by a first trained network at the agent; 
 inferring, at a second trained network at the device, a new label for the unlabeled object based on the one or more labeled object in the first dataset and a second dataset including one or more objects similar to the unlabeled object, the new label being different than a respective label associated each of the one or more labeled objects; 
 label the unlabeled object with the new label to generate a third dataset; and 
 train an untrained network using the third dataset to generate a new model.