Learning Data Augmentation Strategies for Object Detection

Abstract

Example aspects of the present disclosure are directed to systems and methods for learning data augmentation strategies for improved object detection model performance. In particular, example aspects of the present disclosure are directed to iterative reinforcement learning approaches in which, at each of a plurality of iterations, a controller model selects a series of one or more augmentation operations to be applied to training images to generate augmented images. For example, the controller model can select the augmentation operations from a defined search space of available operations which can, for example, include operations that augment the training image without modification of the locations of a target object and corresponding bounding shape within the image and/or operations that do modify the locations of the target object and bounding shape within the training image.

What is claimed is: 
     
         1 - 20 . (canceled) 
     
     
         21 . A computing system comprising:
 one or more processors; and   one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
 accessing a training dataset that comprises a plurality of training images, wherein one or more training images of the plurality of training images has been annotated with a bounding shape, and wherein a location of the bounding shape for the one or more training images corresponds to a location of a target object within the one or more training images; and 
 for each of one or more iterations:
 applying one or more augmentation operations only to the bounding shape of the one or more training images to generate one or more augmented images; 
 training a machine-learned object detection model based at least in part on the one or more augmented images; and 
 after training the machine-learned object detection model, evaluating one or more performance characteristics of the machine-learned object detection model. 
 
   
     
     
         22 . The computing system of  claim 21 , wherein applying the one or more augmentation operations only to the bounding shape of the one or more training images comprises selecting the one or more augmentation operations from a defined search space that includes a plurality of available augmentation operations. 
     
     
         23 . The computing system of  claim 22 , wherein the plurality of available augmentation operations comprises one or more color operations that modify color channel data associated with at least a portion of the one or more training images. 
     
     
         24 . The computing system of  claim 23 , wherein the one or more color operations comprise one or more of:
 an auto contrast operation;   an equalize operation;   a solarize operation;   a posterize operation;   a contrast operation;   a color balance operation;   a brightness operation;   a sharpness operation; and   a cutout operation.   
     
     
         25 . The computing system of  claim 22 , wherein the plurality of available augmentation operations comprises one or more geometric distortion operations that geometrically distort at least a portion of the one or more training images. 
     
     
         26 . The computing system of  claim 25 , wherein the one or more geometric distortion operations comprise one or more of:
 a shear operation;   a translate operation;   a rotate operation;   a flipping operation; and   a shift bounding shape operation.   
     
     
         27 . The computing system of  claim 22 , wherein the plurality of available augmentation operations comprises:
 one or more operations that augment the one or more training images without modification of the location of the bounding shape or the location of the target object within the one or more training images; and   one or more operations that modify the location of the bounding shape and the location of the target object within the one or more training images.   
     
     
         28 . The computing system of  claim 21 , wherein applying the one or more augmentation operations only to the bounding shape of the one or more training images comprises:
 selecting a respective probability of performance for each of the one or more augmentation operations; and   performing each of the one or more augmentation operations with probability equal to the respective probability of performance.   
     
     
         29 . The computing system of  claim 21 , wherein applying the one or more augmentation operations only to the bounding shape of the one or more training images comprises:
 selecting a respective probability that each of the one or more augmentation operations will be applied only to the bounding shape of the one or more training images; and   applying each augmentation operation to only the bounding shape of the one or more training images with probability equal to the respective probability.   
     
     
         30 . The computing system of  claim 21 , wherein applying the one or more augmentation operations only to the bounding shape of the one or more training images comprises:
 selecting a respective augmentation magnitude for at least one of the augmentation operations; and   performing the at least one of the augmentation operations according to the respective augmentation magnitude.   
     
     
         31 . The computing system of  claim 30 , wherein selecting the respective augmentation magnitude for at least one of the augmentation operations comprises selecting the respective augmentation magnitude for at least one of the augmentation operations from a respective set of discrete and operation-specific available magnitudes, wherein the set of discrete and operation-specific available magnitudes comprise user-selected hyperparameters. 
     
     
         32 . The computing system of  claim 21 , wherein the computing system further comprises a controller model comprising a first neural network, and wherein the operations further comprise:
 for each of the one or more iterations:
 determining a reward based at least in part on the one or more performance characteristics; and 
 backpropagating a reward function through the first neural network to modify one or more parameters of the first neural network. 
   
     
     
         33 . The computing system of  claim 21 , wherein the computing system further comprises a controller model comprising a first neural network configured to select one or more augmentation operations through performance of evolutionary mutations, and wherein the operations further comprise, for each of the one or more iterations, determining whether to retain or discard the one or more augmentation operations based at least in part on the one or more performance characteristics. 
     
     
         34 . The computing system of  claim 21 , wherein training the machine-learned object detection model based at least in part on the one or more augmented images comprises:
 evaluating, for each augmented image, a loss function that compares a predicted location for the bounding shape of the augmented image that was predicted by the machine-learned object detection model based on the augmented image to a ground truth location for the bounding shape; and   backpropagating the loss function through the machine-learned object detection model.   
     
     
         35 . The computing system of  claim 21 , wherein, for each iteration, a number of augmentation operations in the one or more augmentation operations is selected by a controller model comprising a first neural network. 
     
     
         36 . The computing system of  claim 35 , wherein the first neural network of the controller model comprises a recurrent neural network. 
     
     
         37 . The computing system of  claim 21 , wherein, for each iteration, a number of augmentation operations in the one or more augmentation operations is a user-selected hyperparameter. 
     
     
         38 . The computing system of  claim 21 , wherein the operations further comprise: 
 obtaining a second training dataset comprising a second plurality of training images;   applying, on one or more second training images of the second plurality of training images, at least one of the one or more augmentation operations only to the bounding shape of the one or more second training images in at least one of the one or more iterations; and   training a second machine-learned object detection model based at least in part on the one or more second training images.   
     
     
         39 . A computer-implemented method, comprising:
 accessing a training dataset that comprises a plurality of training images, wherein one or more training images of the plurality of training images has been annotated with a bounding shape, and wherein a location of the bounding shape for the one or more training images corresponds to a location of a target object within the one or more training images; and   for each of one or more iterations:
 applying one or more augmentation operations only to the bounding shape of the one or more training images to generate one or more augmented images; 
 training a machine-learned object detection model based at least in part on the one or more augmented images; and 
 after training the machine-learned object detection model, evaluating one or more performance characteristics of the machine-learned object detection model. 
   
     
     
         40 . One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
 accessing a training dataset that comprises a plurality of training images, wherein one or more training images of the plurality of training images has been annotated with a bounding shape, and wherein a location of the bounding shape for the one or more training images corresponds to a location of a target object within the one or more training images; and   for each of one or more iterations:
 applying one or more augmentation operations only to the bounding shape of the one or more training images to generate one or more augmented images; 
 training a machine-learned object detection model based at least in part on the one or more augmented images; and 
 after training the machine-learned object detection model, evaluating one or more performance characteristics of the machine-learned object detection model.