A Classical View on Benign Overfitting: The Role of Sample Size

TL;DR

This paper explores how classical statistical regimes can exhibit almost benign overfitting, where models fit training data perfectly yet still generalize well, supported by theoretical analysis of kernel and neural network models.

Contribution

It introduces a new perspective on benign overfitting, demonstrating its emergence in classical regimes through theoretical analysis of kernel ridge regression and neural networks.

Findings

Models achieve low training and test errors simultaneously.

Theoretical evidence supports benign overfitting in classical regimes.

A novel proof technique decomposes excess risk into estimation and approximation errors.

Abstract

Benign overfitting is a phenomenon in machine learning where a model perfectly fits (interpolates) the training data, including noisy examples, yet still generalizes well to unseen data. Understanding this phenomenon has attracted considerable attention in recent years. In this work, we introduce a conceptual shift, by focusing on almost benign overfitting, where models simultaneously achieve both arbitrarily small training and test errors. This behavior is characteristic of neural networks, which often achieve low (but non-zero) training error while still generalizing well. We hypothesize that this almost benign overfitting can emerge even in classical regimes, by analyzing how the interaction between sample size and model complexity enables larger models to achieve both good training fit but still approach Bayes-optimal generalization. We substantiate this hypothesis with theoretical evidence from two case studies: (i) kernel ridge regression, and (ii) least-squares regression using a two-layer fully connected ReLU neural network trained via gradient flow. In both cases, we overcome the strong assumptions often required in prior work on benign overfitting. Our results on neural networks also provide the first generalization result in this setting that does not rely on any assumptions about the underlying regression function or noise, beyond boundedness. Our analysis introduces a novel proof technique based on decomposing the excess risk into estimation and approximation errors, interpreting gradient flow as an implicit regularizer, that helps avoid uniform convergence traps. This analysis idea could be of independent interest.