$\left( \beta, \varpi \right)$-stability for cross-validation and the choice of the number of folds

TL;DR

This paper introduces a new stability concept for cross-validation, called (β, ω)-stability, linking generalization and stability through Rademacher complexity, and provides bounds that inform optimal fold choices for stable, accurate model evaluation.

Contribution

It develops the (β, ω)-stability framework for cross-validation, deriving new Rademacher bounds applicable to both i.i.d. and non-i.i.d. data, and guides optimal fold selection.

Findings

New (β, ω)-stability concept connects generalization and stability.

Derived bounds quantify stability of cross-validation in various settings.

Empirical results suggest optimal number of folds minimizes test error bound.

Abstract

In this paper, we introduce a new concept of stability for cross-validation, called the $\left( \beta, \varpi \right)$-stability, and use it as a new perspective to build the general theory for cross-validation. The $\left( \beta, \varpi \right)$-stability mathematically connects the generalization ability and the stability of the cross-validated model via the Rademacher complexity. Our result reveals mathematically the effect of cross-validation from two sides: on one hand, cross-validation picks the model with the best empirical generalization ability by validating all the alternatives on test sets; on the other hand, cross-validation may compromise the stability of the model selection by causing subsampling error. Moreover, the difference between training and test errors in q\textsuperscript{th} round, sometimes referred to as the generalization error, might be autocorrelated on q. Guided by the ideas above, the $\left( \beta, \varpi \right)$-stability help us derivd a new class of Rademacher bounds, referred to as the one-round/convoluted Rademacher bounds, for the stability of cross-validation in both the i.i.d.\ and non-i.i.d.\ cases. For both light-tail and heavy-tail losses, the new bounds quantify the stability of the one-round/average test error of the cross-validated model in terms of its one-round/average training error, the sample sizes $n$, number of folds $K$, the tail property of the loss (encoded as Orlicz-$\Psi_\nu$ norms) and the Rademacher complexity of the model class $\Lambda$. The new class of bounds not only quantitatively reveals the stability of the generalization ability of the cross-validated model, it also shows empirically the optimal choice for number of folds $K$, at which the upper bound of the one-round/average test error is lowest, or, to put it in another way, where the test error is most stable.