Linearly Convergent Mixup Learning

TL;DR

This paper introduces two new algorithms for mixup data augmentation in RKHS-based binary classification, which are hyperparameter-free, scale linearly, and outperform gradient descent in convergence speed and predictive accuracy.

Contribution

The paper proposes two novel, hyperparameter-free algorithms for mixup learning in RKHS that are more efficient and broadly applicable than existing gradient-based methods.

Findings

Algorithms converge faster than gradient descent.

Mixup improves predictive performance across loss functions.

Methods scale linearly with dataset size.

Abstract

Learning in the reproducing kernel Hilbert space (RKHS) such as the support vector machine has been recognized as a promising technique. It continues to be highly effective and competitive in numerous prediction tasks, particularly in settings where there is a shortage of training data or computational limitations exist. These methods are especially valued for their ability to work with small datasets and their interpretability. To address the issue of limited training data, mixup data augmentation, widely used in deep learning, has remained challenging to apply to learning in RKHS due to the generation of intermediate class labels. Although gradient descent methods handle these labels effectively, dual optimization approaches are typically not directly applicable. In this study, we present two novel algorithms that extend to a broader range of binary classification models. Unlike gradient-based approaches, our algorithms do not require hyperparameters like learning rates, simplifying their implementation and optimization. Both the number of iterations to converge and the computational cost per iteration scale linearly with respect to the dataset size. The numerical experiments demonstrate that our algorithms achieve faster convergence to the optimal solution compared to gradient descent approaches, and that mixup data augmentation consistently improves the predictive performance across various loss functions.