Uniform Consistency of Generalized Cross-Validation for Ridge Regression in High-Dimensional Misspecified Linear Models

TL;DR

This paper proves that generalized cross-validation reliably selects tuning parameters for ridge regression in high-dimensional, potentially misspecified models, ensuring near-optimal prediction performance.

Contribution

It establishes the uniform consistency of generalized cross-validation for ridge regression, even with a broad set of tuning parameters including negative values, under certain misspecification conditions.

Findings

GCV is a consistent estimator of out-of-sample risk.

GCV-tuned ridge matches the performance of optimally tuned ridge.

GCV ridge outperforms Lasso under various model specifications.

Abstract

This study examines generalized cross-validation for the tuning parameter selection for ridge regression in high-dimensional misspecified linear models. The set of candidates for the tuning parameter includes not only positive values but also zero and negative values. We demonstrate that if the second moment of the specification error converges to zero, generalized cross-validation is still a uniformly consistent estimator of the out-of-sample prediction risk. This implies that generalized cross-validation selects the tuning parameter for which ridge regression asymptotically achieves the smallest prediction risk among the candidates if the degree of misspecification for the regression function is small. Our simulation studies show that ridge regression tuned by generalized cross-validation exhibits a prediction performance similar to that of optimally tuned ridge regression and outperforms the Lasso under correct and incorrect model specifications.