Foolish Crowds Support Benign Overfitting

TL;DR

This paper establishes a lower bound on the excess risk of sparse interpolating linear regression procedures in overparameterized Gaussian settings, revealing that basis pursuit can converge more slowly than OLS due to a 'foolish crowd' effect that mitigates noise fitting.

Contribution

It provides the first theoretical lower bound on excess risk for sparse interpolators, highlighting the impact of overparameterization and noise spreading on convergence rates.

Findings

Basis pursuit's excess risk can be exponentially slower than OLS.

Overparameterization and noise spreading reduce variance and improve risk.

The 'foolish crowd' effect explains how fitting noise among many directions benefits interpolation.

Abstract

We prove a lower bound on the excess risk of sparse interpolating procedures for linear regression with Gaussian data in the overparameterized regime. We apply this result to obtain a lower bound for basis pursuit (the minimum $\ell_1$-norm interpolant) that implies that its excess risk can converge at an exponentially slower rate than OLS (the minimum $\ell_2$-norm interpolant), even when the ground truth is sparse. Our analysis exposes the benefit of an effect analogous to the "wisdom of the crowd", except here the harm arising from fitting the $\textit{noise}$ is ameliorated by spreading it among many directions -- the variance reduction arises from a $\textit{foolish}$ crowd.