Double Descent: Understanding Linear Model Estimation of Nonidentifiable Parameters and a Model for Overfitting

TL;DR

This paper investigates the double descent phenomenon in linear models, analyzing estimation methods for high-dimensional settings where the number of parameters exceeds observations, and provides insights into overfitting and prediction accuracy.

Contribution

It offers a comprehensive analysis of double descent in linear models, connecting estimation techniques with overfitting behavior in high-dimensional regimes.

Findings

Double descent occurs in linear models with p > n.

Regularized estimators can mitigate overfitting.

Spectral shrinkage influences prediction performance.

Abstract

We consider ordinary least squares estimation and variations on least squares estimation such as penalized (regularized) least squares and spectral shrinkage estimates for problems with p > n and associated problems with prediction of new observations. After the introduction of Section 1, Section 2 examines a number of commonly used estimators for p > n. Section 3 introduces prediction with p > n. Section 4 introduces notational changes to facilitate discussion of overfitting and Section 5 illustrates the phenomenon of double descent. We conclude with some final comments.