Least Squares Regression Can Exhibit Under-Parameterized Double Descent

TL;DR

This paper demonstrates that least squares regression can show double descent behavior even in under-parameterized regimes, challenging previous assumptions and highlighting the influence of spectral properties.

Contribution

It introduces simple examples where double descent occurs in under-parameterized settings, emphasizing the role of spectral properties of the data covariance.

Findings

Double descent can occur in under-parameterized models.

Spectral properties influence the occurrence of double descent.

Previous explanations do not fully account for these phenomena.

Abstract

The relationship between the number of training data points, the number of parameters, and the generalization capabilities of models has been widely studied. Previous work has shown that double descent can occur in the over-parameterized regime and that the standard bias-variance trade-off holds in the under-parameterized regime. These works provide multiple reasons for the existence of the peak. We postulate that the location of the peak depends on the technical properties of both the spectrum as well as the eigenvectors of the sample covariance. We present two simple examples that provably exhibit double descent in the under-parameterized regime and do not seem to occur for reasons provided in prior work.