Benign overfitting without concentration

TL;DR

This paper provides a new sufficient condition for benign overfitting in linear regression that does not depend on concentration properties, making it applicable to heavy-tailed data.

Contribution

It introduces a small-ball assumption-based approach for benign overfitting analysis, bypassing reliance on concentration arguments and extending applicability to heavy-tailed distributions.

Findings

Benign overfitting can occur without concentration assumptions.

The method applies to heavy-tailed data distributions.

A new small-ball estimate in terms of effective rank is established.

Abstract

We obtain a sufficient condition for benign overfitting of linear regression problem. Our result does not rely on concentration argument but on small-ball assumption and thus can holds in heavy-tailed case. The basic idea is to establish a coordinate small-ball estimate in terms of effective rank so that we can calibrate the balance of epsilon-Net and exponential probability. Our result indicates that benign overfitting is not depending on concentration property of the input vector. Finally, we discuss potential difficulties for benign overfitting beyond linear model and a benign overfitting result without truncated effective rank.