On Optimal Interpolation In Linear Regression

TL;DR

This paper derives the optimal linear response-interpolating estimator in linear regression, showing how it can outperform the minimum-norm interpolator and extending the analysis to random features regression.

Contribution

It provides a closed-form expression for the optimal interpolator in linear regression and connects it to gradient descent limits, advancing understanding of interpolation and generalization.

Findings

Optimal interpolator derived in closed form.

Minimum-norm interpolator can generalize worse than the optimal.

Extension of optimal interpolation concept to random features regression.

Abstract

Understanding when and why interpolating methods generalize well has recently been a topic of interest in statistical learning theory. However, systematically connecting interpolating methods to achievable notions of optimality has only received partial attention. In this paper, we investigate the question of what is the optimal way to interpolate in linear regression using functions that are linear in the response variable (as the case for the Bayes optimal estimator in ridge regression) and depend on the data, the population covariance of the data, the signal-to-noise ratio and the covariance of the prior for the signal, but do not depend on the value of the signal itself nor the noise vector in the training data. We provide a closed-form expression for the interpolator that achieves this notion of optimality and show that it can be derived as the limit of preconditioned gradient descent with a specific initialization. We identify a regime where the minimum-norm interpolator provably generalizes arbitrarily worse than the optimal response-linear achievable interpolator that we introduce, and validate with numerical experiments that the notion of optimality we consider can be achieved by interpolating methods that only use the training data as input in the case of an isotropic prior. Finally, we extend the notion of optimal response-linear interpolation to random features regression under a linear data-generating model that has been previously studied in the literature.