From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators

TL;DR

This paper derives the asymptotic risk function of regularized estimators tuned by cross-validation, showing convergence to SURE and providing detailed insights into predictive performance in high-dimensional settings.

Contribution

It establishes the asymptotic equivalence between cross-validation tuned estimators and SURE-tuned estimators, with detailed analysis of convergence and risk behavior.

Findings

CV converges uniformly to SURE

SURE's global minimum is well separated

Risk varies with the true parameter

Abstract

We derive the asymptotic risk function of regularized empirical risk minimization (ERM) estimators tuned by $n$-fold cross-validation (CV). The out-of-sample prediction loss of such estimators converges in distribution to the squared-error loss (risk function) of shrinkage estimators in the normal means model, tuned by Stein's unbiased risk estimate (SURE). This risk function provides a more fine-grained picture of predictive performance than uniform bounds on worst-case regret, which are common in learning theory: it quantifies how risk varies with the true parameter. As key intermediate steps, we show that (i) $n$-fold CV converges uniformly to SURE, and (ii) while SURE typically has multiple local minima, its global minimum is generically well separated. Well-separation ensures that uniform convergence of CV to SURE translates into convergence of the tuning parameter chosen by CV to that chosen by SURE.