Harmful Overfitting in Sobolev Spaces

TL;DR

This paper investigates how functions in Sobolev spaces that perfectly fit noisy data can still generalize poorly, showing that norm-minimizing interpolators suffer from harmful overfitting even with large sample sizes.

Contribution

It extends the understanding of overfitting in Sobolev spaces for all p in [1, ∞), beyond the Hilbert space case, using geometric Sobolev inequalities.

Findings

Norm-minimizing interpolators exhibit persistent generalization error.

Harmful overfitting occurs despite increasing sample size.

Results apply to a broad range of Sobolev spaces, not just Hilbert spaces.

Abstract

Motivated by recent work on benign overfitting in overparameterized machine learning, we study the generalization behavior of functions in Sobolev spaces $W^{k, p}(\mathbb{R}^d)$ that perfectly fit a noisy training data set. Under assumptions of label noise and sufficient regularity in the data distribution, we show that approximately norm-minimizing interpolators, which are canonical solutions selected by smoothness bias, exhibit harmful overfitting: even as the training sample size $n \to \infty$, the generalization error remains bounded below by a positive constant with high probability. Our results hold for arbitrary values of $p \in [1, \infty)$, in contrast to prior results studying the Hilbert space case ($p = 2$) using kernel methods. Our proof uses a geometric argument which identifies harmful neighborhoods of the training data using Sobolev inequalities.