Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regression

TL;DR

This paper introduces an 'optimistic rate' framework for analyzing linear regression with Gaussian data, unifying understanding of interpolation learning and regularization, and providing refined bounds that improve high-dimensional risk assessments.

Contribution

It presents a refined analysis of optimistic rates that removes hidden constants and logarithmic factors, unifying and extending guarantees for interpolators, ridge, and LASSO regressions.

Findings

Refined bounds for population risk of low-norm interpolators.

Unified analysis applicable to predictors with arbitrary training error.

Enhanced understanding of excess risk in over-parameterized models.

Abstract

We study a localized notion of uniform convergence known as an "optimistic rate" (Panchenko 2002; Srebro et al. 2010) for linear regression with Gaussian data. Our refined analysis avoids the hidden constant and logarithmic factor in existing results, which are known to be crucial in high-dimensional settings, especially for understanding interpolation learning. As a special case, our analysis recovers the guarantee from Koehler et al. (2021), which tightly characterizes the population risk of low-norm interpolators under the benign overfitting conditions. Our optimistic rate bound, though, also analyzes predictors with arbitrary training error. This allows us to recover some classical statistical guarantees for ridge and LASSO regression under random designs, and helps us obtain a precise understanding of the excess risk of near-interpolators in the over-parameterized regime.