Memorize to Generalize: on the Necessity of Interpolation in High Dimensional Linear Regression

TL;DR

This paper demonstrates that in high-dimensional linear regression, achieving optimal prediction accuracy necessitates near-perfect interpolation of training data, especially as noise diminishes, highlighting the fundamental role of interpolation in overparameterized models.

Contribution

It provides a precise characterization of the relationship between training and test error in overparameterized linear regression, emphasizing the necessity of interpolation for optimal performance.

Findings

Prediction error scales with training error in high-dimensional linear models.

Optimal performance requires fitting training data beyond the noise level.

Suboptimal estimators incur excess prediction error proportional to training error.

Abstract

We examine the necessity of interpolation in overparameterized models, that is, when achieving optimal predictive risk in machine learning problems requires (nearly) interpolating the training data. In particular, we consider simple overparameterized linear regression $y = X \theta + w$ with random design $X \in \mathbb{R}^{n \times d}$ under the proportional asymptotics $d/n \to \gamma \in (1, \infty)$. We precisely characterize how prediction (test) error necessarily scales with training error in this setting. An implication of this characterization is that as the label noise variance $\sigma^2 \to 0$, any estimator that incurs at least $\mathsf{c}\sigma^4$ training error for some constant $\mathsf{c}$ is necessarily suboptimal and will suffer growth in excess prediction error at least linear in the training error. Thus, optimal performance requires fitting training data to substantially higher accuracy than the inherent noise floor of the problem.