Provable Benefit of Mixup for Finding Optimal Decision Boundaries

TL;DR

This paper demonstrates that Mixup data augmentation reduces the sample complexity in finding optimal decision boundaries in binary classification, especially for highly separable data, and analyzes its theoretical benefits and limitations.

Contribution

The paper provides a theoretical analysis of Mixup's benefit in reducing sample complexity and introduces new concentration results for pair-wise augmented data.

Findings

Mixup mitigates the curse of separability by reducing sample complexity.

Vanilla training's sample complexity increases exponentially with data separability.

Other masking-based Mixup techniques can distort training loss and lead to suboptimal classifiers.

Abstract

We investigate how pair-wise data augmentation techniques like Mixup affect the sample complexity of finding optimal decision boundaries in a binary linear classification problem. For a family of data distributions with a separability constant $\kappa$, we analyze how well the optimal classifier in terms of training loss aligns with the optimal one in test accuracy (i.e., Bayes optimal classifier). For vanilla training without augmentation, we uncover an interesting phenomenon named the curse of separability. As we increase $\kappa$ to make the data distribution more separable, the sample complexity of vanilla training increases exponentially in $\kappa$; perhaps surprisingly, the task of finding optimal decision boundaries becomes harder for more separable distributions. For Mixup training, we show that Mixup mitigates this problem by significantly reducing the sample complexity. To this end, we develop new concentration results applicable to $n^2$ pair-wise augmented data points constructed from $n$ independent data, by carefully dealing with dependencies between overlapping pairs. Lastly, we study other masking-based Mixup-style techniques and show that they can distort the training loss and make its minimizer converge to a suboptimal classifier in terms of test accuracy.