Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Bias

TL;DR

This paper investigates how strong inductive biases in noisy interpolation models can both help and hinder generalization, revealing that certain norms achieve faster convergence rates than others, with implications for linear and non-linear models.

Contribution

It demonstrates that for noisy data, minimum $ ext{l}_p$-norm and maximum margin interpolators with $p > 1$ attain near $1/n$ rates, challenging conventional wisdom about inductive bias effects.

Findings

$ ext{l}_p$-norm interpolators with $p > 1$ achieve near $1/n$ rates.

Strong inductive bias can increase noise effects despite promoting simplicity.

Preliminary experiments suggest similar trade-offs in non-linear models.

Abstract

Good generalization performance on high-dimensional data crucially hinges on a simple structure of the ground truth and a corresponding strong inductive bias of the estimator. Even though this intuition is valid for regularized models, in this paper we caution against a strong inductive bias for interpolation in the presence of noise: While a stronger inductive bias encourages a simpler structure that is more aligned with the ground truth, it also increases the detrimental effect of noise. Specifically, for both linear regression and classification with a sparse ground truth, we prove that minimum $\ell_p$-norm and maximum $\ell_p$-margin interpolators achieve fast polynomial rates close to order $1/n$ for $p > 1$ compared to a logarithmic rate for $p = 1$. Finally, we provide preliminary experimental evidence that this trade-off may also play a crucial role in understanding non-linear interpolating models used in practice.