Benign Overfitting in Linear Classifiers with a Bias Term

TL;DR

This paper extends the understanding of benign overfitting in linear classifiers by analyzing models with a bias term, revealing how it influences generalization conditions especially in noisy and isotropic data settings.

Contribution

It generalizes previous homogeneous model results to inhomogeneous models with bias, showing how bias affects benign overfitting and generalization constraints.

Findings

Benign overfitting persists with a bias term in linear models.

Bias introduces new covariance constraints for generalization.

In isotropic cases, these constraints are dominated by homogeneous model requirements.

Abstract

Modern machine learning models with a large number of parameters often generalize well despite perfectly interpolating noisy training data - a phenomenon known as benign overfitting. A foundational explanation for this in linear classification was recently provided by Hashimoto et al. (2025). However, this analysis was limited to the setting of "homogeneous" models, which lack a bias (intercept) term - a standard component in practice. This work directly extends Hashimoto et al.'s results to the more realistic inhomogeneous case, which incorporates a bias term. Our analysis proves that benign overfitting persists in these more complex models. We find that the presence of the bias term introduces new constraints on the data's covariance structure required for generalization, an effect that is particularly pronounced when label noise is present. However, we show that in the isotropic case, these new constraints are dominated by the requirements inherited from the homogeneous model. This work provides a more complete picture of benign overfitting, revealing the non-trivial impact of the bias term on the conditions required for good generalization.