Ridge regression and asymptotic minimax estimation over spheres of growing dimension

TL;DR

This paper demonstrates that ridge regression is asymptotically minimax for estimating high-dimensional parameters over growing spheres under Gaussian linear models, providing new formulas for its risk and proposing adaptive estimators.

Contribution

It establishes ridge regression as asymptotically minimax in high-dimensional settings and derives explicit risk formulas involving spectral distributions, also proposing adaptive estimators.

Findings

Ridge regression is asymptotically minimax for high-dimensional estimation.

Explicit risk formulas involve the Marčenko-Pastur distribution.

Adaptive ridge estimators effectively handle unknown sphere radii.

Abstract

We study asymptotic minimax problems for estimating a $d$-dimensional regression parameter over spheres of growing dimension ($d\to \infty$). Assuming that the data follows a linear model with Gaussian predictors and errors, we show that ridge regression is asymptotically minimax and derive new closed form expressions for its asymptotic risk under squared-error loss. The asymptotic risk of ridge regression is closely related to the Stieltjes transform of the Mar\v{c}enko-Pastur distribution and the spectral distribution of the predictors from the linear model. Adaptive ridge estimators are also proposed (which adapt to the unknown radius of the sphere) and connections with equivariant estimation are highlighted. Our results are mostly relevant for asymptotic settings where the number of observations, $n$, is proportional to the number of predictors, that is, $d/n\to\rho\in(0,\infty)$.