A Regularization-Sharpness Tradeoff for Linear Interpolators

TL;DR

This paper introduces a regularization-sharpness tradeoff framework for overparameterized linear regression, analyzing how different penalties influence interpolator performance and generalization, supported by empirical validation.

Contribution

It extends the interpolating information criterion to  and  regularizers, providing a new theoretical understanding of the tradeoff in overparameterized models.

Findings

The tradeoff distinguishes effective interpolators from weaker ones.

The framework applies to  and  regularized interpolators.

Empirical results validate the theoretical predictions.

Abstract

The rule of thumb regarding the relationship between the bias-variance tradeoff and model size plays a key role in classical machine learning, but is now well-known to break down in the overparameterized setting as per the double descent curve. In particular, minimum-norm interpolating estimators can perform well, suggesting the need for new tradeoff in these settings. Accordingly, we propose a regularization-sharpness tradeoff for overparameterized linear regression with an $\ell^p$ penalty. Inspired by the interpolating information criterion, our framework decomposes the selection penalty into a regularization term (quantifying the alignment of the regularizer and the interpolator) and a geometric sharpness term on the interpolating manifold (quantifying the effect of local perturbations), yielding a tradeoff analogous to bias-variance. Building on prior analyses that established this information criterion for ridge regularizers, this work first provides a general expression of the interpolating information criterion for $\ell^p$ regularizers where $p \ge 2$. Subsequently, we extend this to the LASSO interpolator with $\ell^1$ regularizer, which induces stronger sparsity. Empirical results on real-world datasets with random Fourier features and polynomials validate our theory, demonstrating how the tradeoff terms can distinguish performant linear interpolators from weaker ones.