Extrapolated cross-validation for randomized ensembles

TL;DR

This paper introduces ECV, an extrapolated cross-validation method for efficiently tuning ensemble and subsample sizes in randomized ensemble methods, providing theoretical guarantees and practical advantages over traditional CV methods.

Contribution

The paper presents a novel risk extrapolation technique for tuning ensemble parameters, with proven consistency and applicability to high-dimensional settings.

Findings

ECV achieves higher accuracy than traditional CV methods.

ECV reduces computational cost compared to sample-split CV.

Theoretical guarantees hold for high-dimensional regimes.

Abstract

Ensemble methods such as bagging and random forests are ubiquitous in various fields, from finance to genomics. Despite their prevalence, the question of the efficient tuning of ensemble parameters has received relatively little attention. This paper introduces a cross-validation method, ECV (Extrapolated Cross-Validation), for tuning the ensemble and subsample sizes in randomized ensembles. Our method builds on two primary ingredients: initial estimators for small ensemble sizes using out-of-bag errors and a novel risk extrapolation technique that leverages the structure of prediction risk decomposition. By establishing uniform consistency of our risk extrapolation technique over ensemble and subsample sizes, we show that ECV yields $\delta$-optimal (with respect to the oracle-tuned risk) ensembles for squared prediction risk. Our theory accommodates general ensemble predictors, only requires mild moment assumptions, and allows for high-dimensional regimes where the feature dimension grows with the sample size. As a practical case study, we employ ECV to predict surface protein abundances from gene expressions in single-cell multiomics using random forests. In comparison to sample-split cross-validation and $K$-fold cross-validation, ECV achieves higher accuracy avoiding sample splitting. At the same time, its computational cost is considerably lower owing to the use of the risk extrapolation technique. Additional numerical results validate the finite-sample accuracy of ECV for several common ensemble predictors under a computational constraint on the maximum ensemble size.