Bootstrapping the Out-of-sample Predictions for Efficient and Accurate Cross-Validation

TL;DR

This paper introduces two bootstrap-based methods, BBC-CV and BCED-CV, that improve the efficiency and accuracy of cross-validation for model selection and performance estimation by correcting bias and speeding up the process.

Contribution

The paper presents novel bootstrap techniques for bias correction and early stopping in cross-validation, enhancing efficiency and accuracy over existing methods.

Findings

BBC-CV reduces bias and variance in performance estimates.

BCED-CV accelerates cross-validation by early dropping of inferior configurations.

Methods are applicable to various performance metrics.

Abstract

Cross-Validation (CV), and out-of-sample performance-estimation protocols in general, are often employed both for (a) selecting the optimal combination of algorithms and values of hyper-parameters (called a configuration) for producing the final predictive model, and (b) estimating the predictive performance of the final model. However, the cross-validated performance of the best configuration is optimistically biased. We present an efficient bootstrap method that corrects for the bias, called Bootstrap Bias Corrected CV (BBC-CV). BBC-CV's main idea is to bootstrap the whole process of selecting the best-performing configuration on the out-of-sample predictions of each configuration, without additional training of models. In comparison to the alternatives, namely the nested cross-validation and a method by Tibshirani and Tibshirani, BBC-CV is computationally more efficient, has smaller variance and bias, and is applicable to any metric of performance (accuracy, AUC, concordance index, mean squared error). Subsequently, we employ again the idea of bootstrapping the out-of-sample predictions to speed up the CV process. Specifically, using a bootstrap-based hypothesis test we stop training of models on new folds of statistically-significantly inferior configurations. We name the method Bootstrap Corrected with Early Dropping CV (BCED-CV) that is both efficient and provides accurate performance estimates.