A statistical theory of overfitting for imbalanced classification

TL;DR

This paper develops a high-dimensional statistical theory explaining overfitting in imbalanced classification, revealing how class imbalance and dimensionality skew logits and affect minority class performance.

Contribution

It introduces a novel high-dimensional asymptotic framework for analyzing overfitting in imbalanced data, highlighting the impact of dimensionality and class imbalance on model logits.

Findings

Logits for minority classes follow a rectified normal distribution on training data.

Margin rebalancing improves minority class accuracy.

Overfitting affects calibration and uncertainty measures.

Abstract

Classification with imbalanced data is a common challenge in data analysis, where certain classes (minority classes) account for a small fraction of the training data compared with other classes (majority classes). Classical statistical theory based on large-sample asymptotics and finite-sample corrections is often ineffective for high-dimensional data, leaving many overfitting phenomena in empirical machine learning unexplained. In this paper, we develop a statistical theory for high-dimensional imbalanced classification by investigating support vector machines and logistic regression. We find that dimensionality induces truncation or skewing effects on the logit distribution, which we characterize via a variational problem under high-dimensional asymptotics. In particular, for linearly separable data generated from a two-component Gaussian mixture model, the logits from each class follow a normal distribution $\mathsf{N}(0,1)$ on the testing set, but asymptotically follow a rectified normal distribution $\max\{\kappa, \mathsf{N}(0,1)\}$ on the training set -- which is a pervasive phenomenon we verified on tabular data, image data, and text data. This phenomenon explains why the minority class is more severely affected by overfitting. Further, we show that margin rebalancing, which incorporates class sizes into the loss function, is crucial for mitigating the accuracy drop for the minority class. Our theory also provides insights into the effects of overfitting on calibration and other uncertain quantification measures.