Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validation

TL;DR

This paper analyzes subsampling ridge ensembles in high-dimensional settings, characterizing their risk and showing GCV can effectively tune these models to match optimal ridge predictor performance.

Contribution

It provides a theoretical analysis of subsampling ridge ensembles, establishing risk equivalences and demonstrating the effectiveness of GCV for tuning in high-dimensional regimes.

Findings

Optimal ensemble risk matches full ridge predictor risk.

GCV provides consistent risk estimation across subsample sizes.

Tuning without sample splitting achieves optimal ridge performance.

Abstract

We study subsampling-based ridge ensembles in the proportional asymptotics regime, where the feature size grows proportionally with the sample size such that their ratio converges to a constant. By analyzing the squared prediction risk of ridge ensembles as a function of the explicit penalty $\lambda$ and the limiting subsample aspect ratio $\phi_s$ (the ratio of the feature size to the subsample size), we characterize contours in the $(\lambda, \phi_s)$-plane at any achievable risk. As a consequence, we prove that the risk of the optimal full ridgeless ensemble (fitted on all possible subsamples) matches that of the optimal ridge predictor. In addition, we prove strong uniform consistency of generalized cross-validation (GCV) over the subsample sizes for estimating the prediction risk of ridge ensembles. This allows for GCV-based tuning of full ridgeless ensembles without sample splitting and yields a predictor whose risk matches optimal ridge risk.