The asymptotic effect of tuning parameters

TL;DR

This paper develops a detailed asymptotic theory for tuning parameters in estimation procedures, accounting for their data-driven selection via risk minimization and cross-validation.

Contribution

It provides the first comprehensive asymptotic analysis of tuning parameter selection, including consistency, asymptotic normality, and variance estimation.

Findings

Optimal estimators converge to a well-defined limit.

Asymptotic normality holds after proper scaling.

Explicit formulas for the limiting variance are derived.

Abstract

Tuning parameters are parameters involved in an estimating procedure for the purpose of reducing the risk of some other estimator. Examples include the degree of penalization in penalized regression and likelihood problems, as well as the balance parameter in hybrid methods. Typically tuning parameters are set to the minimizers of some estimator of the risk, a step which introduces additional randomness and makes standard methodology inapplicable. We derive precise asymptotic theory for this situation. Our framework allows for smooth, but otherwise arbitrary, loss functions and for the risk to be estimated by cross-validation procedures. Results include consistency of the optimal estimator towards a well-defined quantity and asymptotic normality after proper scaling and centring. We give explicit forms and estimators for the limiting variance matrix and results sharply characterizing the distance from the training error to the cross-validated estimator of the risk.