AutoDO: Robust AutoAugment for Biased Data with Label Noise via Scalable Probabilistic Implicit Differentiation

TL;DR

AutoDO introduces a robust automated dataset optimization framework that improves deep learning model generalization on biased and noisy data by explicitly estimating distribution-changing hyperparameters through scalable implicit differentiation.

Contribution

It reformulates AutoAugment as a dataset optimization task with explicit hyperparameter estimation, enhancing robustness to bias and label noise.

Findings

Up to 9.3% accuracy improvement on biased datasets.

Up to 36.6% gain for underrepresented classes.

Scales linearly with dataset size using Fisher information.

Abstract

AutoAugment has sparked an interest in automated augmentation methods for deep learning models. These methods estimate image transformation policies for train data that improve generalization to test data. While recent papers evolved in the direction of decreasing policy search complexity, we show that those methods are not robust when applied to biased and noisy data. To overcome these limitations, we reformulate AutoAugment as a generalized automated dataset optimization (AutoDO) task that minimizes the distribution shift between test data and distorted train dataset. In our AutoDO model, we explicitly estimate a set of per-point hyperparameters to flexibly change distribution of train data. In particular, we include hyperparameters for augmentation, loss weights, and soft-labels that are jointly estimated using implicit differentiation. We develop a theoretical probabilistic interpretation of this framework using Fisher information and show that its complexity scales linearly with the dataset size. Our experiments on SVHN, CIFAR-10/100, and ImageNet classification show up to 9.3% improvement for biased datasets with label noise compared to prior methods and, importantly, up to 36.6% gain for underrepresented SVHN classes.