Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?

TL;DR

This paper investigates the bias introduced when using Stein's unbiased risk estimator (SURE) to tune estimators, revealing that the apparent error estimate systematically underestimates the true prediction error.

Contribution

It formally characterizes the downward bias of SURE when used for tuning, highlighting limitations of SURE-based model selection.

Findings

SURE tends to underestimate the true prediction error after tuning.

The bias of SURE increases with the complexity of the estimator.

The paper provides theoretical insights into the bias behavior of SURE in model selection.

Abstract

Nearly all estimators in statistical prediction come with an associated tuning parameter, in one way or another. Common practice, given data, is to choose the tuning parameter value that minimizes a constructed estimate of the prediction error of the estimator; we focus on Stein's unbiased risk estimator, or SURE (Stein, 1981; Efron, 1986) which forms an unbiased estimate of the prediction error by augmenting the observed training error with an estimate of the degrees of freedom of the estimator. Parameter tuning via SURE minimization has been advocated by many authors, in a wide variety of problem settings, and in general, it is natural to ask: what is the prediction error of the SURE-tuned estimator? An obvious strategy would be simply use the apparent error estimate as reported by SURE, i.e., the value of the SURE criterion at its minimum, to estimate the prediction error of the SURE-tuned estimator. But this is no longer unbiased; in fact, we would expect that the minimum of the SURE criterion is systematically biased downwards for the true prediction error. In this paper, we formally describe and study this bias.