On the Asymptotic Optimality of Cross-Validation based Hyper-parameter Estimators for Regularized Least Squares Regression Problems

TL;DR

This paper investigates the asymptotic optimality of various cross-validation hyper-parameter estimators in regularized least squares regression, showing that some CV methods are asymptotically optimal while others are not.

Contribution

It provides a theoretical analysis of the asymptotic optimality of several CV-based hyper-parameter estimators in fixed-parameter regression problems.

Findings

Leave k-out, generalized, and r-fold CV are asymptotically optimal under mild assumptions.

Hold out CV is not asymptotically optimal.

Monte Carlo simulations support the theoretical results.

Abstract

The asymptotic optimality (a.o.) of various hyper-parameter estimators with different optimality criteria has been studied in the literature for regularized least squares regression problems. The estimators include e.g., the maximum (marginal) likelihood method, $C_p$ statistics, and generalized cross validation method, and the optimality criteria are based on e.g., the inefficiency, the expectation inefficiency and the risk. In this paper, we consider the regularized least squares regression problems with fixed number of regression parameters, choose the optimality criterion based on the risk, and study the a.o. of several cross validation (CV) based hyper-parameter estimators including the leave $k$-out CV method, generalized CV method, $r$-fold CV method and hold out CV method. We find the former three methods can be a.o. under mild assumptions, but not the last one, and we use Monte Carlo simulations to illustrate the efficacy of our findings.