A geometrical viewpoint on the benign overfitting property of the minimum $l_2$-norm interpolant estimator and its universality

TL;DR

This paper offers a geometric analysis of the benign overfitting phenomenon in minimum l2-norm interpolant estimators, demonstrating their universality and extending convergence results beyond traditional assumptions.

Contribution

It introduces a novel geometric framework using Dvoretsky-Milman theorem to analyze benign overfitting, extending results to heavy-tailed data scenarios.

Findings

Benign overfitting is universal across different data distributions.

The geometric approach improves convergence rate estimates.

Heavy-tailed data scenarios also exhibit benign overfitting.

Abstract

In the linear regression model, the minimum l2-norm interpolant estimator has received much attention since it was proved to be consistent even though it fits noisy data perfectly under some condition on the covariance matrix $\Sigma$ of the input vector, known as benign overfitting. Motivated by this phenomenon, we study the generalization property of this estimator from a geometrical viewpoint. Our main results extend and improve the convergence rates as well as the deviation probability from [Tsigler and Bartlett]. Our proof differs from the classical bias/variance analysis and is based on the self-induced regularization property introduced in [Bartlett, Montanari and Rakhlin]: the minimum l2-norm interpolant estimator can be written as a sum of a ridge estimator and an overfitting component. The two geometrical properties of random Gaussian matrices at the heart of our analysis are the Dvoretsky-Milman theorem and isomorphic and restricted isomorphic properties. In particular, the Dvoretsky dimension appearing naturally in our geometrical viewpoint, coincides with the effective rank and is the key tool for handling the behavior of the design matrix restricted to the sub-space where overfitting happens. We extend these results to heavy-tailed scenarii proving the universality of this phenomenon beyond exponential moment assumptions. This phenomenon is unknown before and is widely believed to be a significant challenge. This follows from an anistropic version of the probabilistic Dvoretsky-Milman theorem that holds for heavy-tailed vectors which is of independent interest.