Benign Overfitting under Learning Rate Conditions for $\alpha$ Sub-exponential Input

TL;DR

This paper extends the analysis of benign overfitting to heavy-tailed input distributions, providing generalization bounds and showing that overfitting can persist even with very heavy tails, under certain conditions.

Contribution

It introduces a theoretical framework for benign overfitting under $eta$ learning rate conditions for $ ext{alpha}$ sub-exponential distributions, generalizing prior sub-gaussian results.

Findings

Benign overfitting persists in heavy-tailed input settings.

Generalization error approaches the noise level under certain conditions.

Upper bounds on the learning rate decrease as tail heaviness increases.

Abstract

This paper investigates the phenomenon of benign overfitting in binary classification problems with heavy-tailed input distributions, extending the analysis of maximum margin classifiers to $\alpha$ sub-exponential distributions ($\alpha \in (0, 2]$). This generalizes previous work focused on sub-gaussian inputs. We provide generalization error bounds for linear classifiers trained using gradient descent on unregularized logistic loss in this heavy-tailed setting. Our results show that, under certain conditions on the dimensionality $p$ and the distance between the centers of the distributions, the misclassification error of the maximum margin classifier asymptotically approaches the noise level, the theoretical optimal value. Moreover, we derive an upper bound on the learning rate $\beta$ for benign overfitting to occur and show that as the tail heaviness of the input distribution $\alpha$ increases, the upper bound on the learning rate decreases. These results demonstrate that benign overfitting persists even in settings with heavier-tailed inputs than previously studied, contributing to a deeper understanding of the phenomenon in more realistic data environments.