On the bias of K-fold cross validation with stable learners

TL;DR

This paper examines the bias in K-fold cross-validation when used with stable learning algorithms, showing that standard estimates can be inconsistent and proposing a debiased alternative with proven error bounds.

Contribution

It introduces a debiased K-fold cross-validation method applicable to a broad class of stable algorithms, with theoretical error bounds and empirical validation.

Findings

Standard K-fold CV can be biased and inconsistent for stable learners.

Debiased K-fold CV provides more accurate risk estimates with exponential tail bounds.

Empirical results demonstrate improved model selection performance on real datasets.

Abstract

This paper investigates the efficiency of the K-fold cross-validation (CV) procedure and a debiased version thereof as a means of estimating the generalization risk of a learning algorithm. We work under the general assumption of uniform algorithmic stability. We show that the K-fold risk estimate may not be consistent under such general stability assumptions, by constructing non vanishing lower bounds on the error in realistic contexts such as regularized empirical risk minimisation and stochastic gradient descent. We thus advocate the use of a debiased version of the K-fold and prove an error bound with exponential tail decay regarding this version. Our result is applicable to the large class of uniformly stable algorithms, contrarily to earlier works focusing on specific tasks such as density estimation. We illustrate the relevance of the debiased K-fold CV on a simple model selection problem and demonstrate empirically the usefulness of the promoted approach on real world classification and regression datasets.