When to Learn What: Model-Adaptive Data Augmentation Curriculum

TL;DR

This paper introduces MADAug, a model-adaptive data augmentation method that dynamically learns when and how to augment data during training, leading to improved generalization and fairness across classes.

Contribution

MADAug jointly trains an augmentation policy network with the model using bi-level optimization, creating an adaptive curriculum tailored to each sample and training stage.

Findings

MADAug outperforms existing data augmentation methods on multiple image classification tasks.

It improves fairness by benefitting difficult classes more.

The learned policy transfers well to fine-grained datasets.

Abstract

Data augmentation (DA) is widely used to improve the generalization of neural networks by enforcing the invariances and symmetries to pre-defined transformations applied to input data. However, a fixed augmentation policy may have different effects on each sample in different training stages but existing approaches cannot adjust the policy to be adaptive to each sample and the training model. In this paper, we propose Model Adaptive Data Augmentation (MADAug) that jointly trains an augmentation policy network to teach the model when to learn what. Unlike previous work, MADAug selects augmentation operators for each input image by a model-adaptive policy varying between training stages, producing a data augmentation curriculum optimized for better generalization. In MADAug, we train the policy through a bi-level optimization scheme, which aims to minimize a validation-set loss of a model trained using the policy-produced data augmentations. We conduct an extensive evaluation of MADAug on multiple image classification tasks and network architectures with thorough comparisons to existing DA approaches. MADAug outperforms or is on par with other baselines and exhibits better fairness: it brings improvement to all classes and more to the difficult ones. Moreover, MADAug learned policy shows better performance when transferred to fine-grained datasets. In addition, the auto-optimized policy in MADAug gradually introduces increasing perturbations and naturally forms an easy-to-hard curriculum.