MixUp as Locally Linear Out-Of-Manifold Regularization

TL;DR

This paper analyzes MixUp as a form of out-of-manifold regularization, identifies a limitation called manifold intrusion, and proposes AdaMixUp, an adaptive version that learns mixing policies to improve deep neural network training.

Contribution

It provides a theoretical understanding of MixUp, introduces the concept of manifold intrusion, and proposes AdaMixUp, an adaptive regularizer that enhances MixUp's effectiveness.

Findings

AdaMixUp outperforms standard MixUp on benchmark datasets.

Manifold intrusion can cause under-fitting in MixUp.

Adaptive learning of mixing policies mitigates manifold intrusion.

Abstract

MixUp is a recently proposed data-augmentation scheme, which linearly interpolates a random pair of training examples and correspondingly the one-hot representations of their labels. Training deep neural networks with such additional data is shown capable of significantly improving the predictive accuracy of the current art. The power of MixUp, however, is primarily established empirically and its working and effectiveness have not been explained in any depth. In this paper, we develop an understanding for MixUp as a form of "out-of-manifold regularization", which imposes certain "local linearity" constraints on the model's input space beyond the data manifold. This analysis enables us to identify a limitation of MixUp, which we call "manifold intrusion". In a nutshell, manifold intrusion in MixUp is a form of under-fitting resulting from conflicts between the synthetic labels of the mixed-up examples and the labels of original training data. Such a phenomenon usually happens when the parameters controlling the generation of mixing policies are not sufficiently fine-tuned on the training data. To address this issue, we propose a novel adaptive version of MixUp, where the mixing policies are automatically learned from the data using an additional network and objective function designed to avoid manifold intrusion. The proposed regularizer, AdaMixUp, is empirically evaluated on several benchmark datasets. Extensive experiments demonstrate that AdaMixUp improves upon MixUp when applied to the current art of deep classification models.