Uncertainty quantification via cross-validation and its variants under algorithmic stability

TL;DR

This paper establishes asymptotic guarantees for cross-validation (CV) methods in uncertainty quantification, demonstrating their conservative coverage properties under minimal assumptions and extending results to CV+ in high-dimensional settings.

Contribution

It proves asymptotic conditional conservativeness of CV under minimal assumptions and shows CV+ shares the same guarantees as CV in large samples, extending applicability.

Findings

CV is asymptotically conservatively calibrated conditional on training data.

CV+ provides the same guarantees as CV in large samples.

CV+ improves over simple CV in terms of marginal coverage.

Abstract

Recently, there has been substantial interest in statistical guarantees for cross-validation (CV) methods of uncertainty quantification in statistical learning (cf. Barber et al. 2021a, Liang and Barber 2024, Steinberger and Leeb 2023). These guarantees should hold under minimal assumptions on the data generating process and conditional on the training data, because numerous predictions are usually computed based on one and the same training sample. We push this objective to the limit: We prove asymptotic conditional conservativeness of CV, that is, the probability of the actual coverage probability, conditional on the training data, undershooting its nominal level vanishes asymptotically, under minimal assumptions. In particular, we impose a stability condition, require that the prediction error is stochastically bounded, and show that neither condition can be dropped in general. By way of an asymptotic equivalence result, we also show that the closely related CV+ method of Barber et al. (2021a) provides exactly the same conditional statistical guarantees as CV in large samples, thereby extending the range of applicability of CV+ to the high-dimensional regime. We conclude that, in view of its marginal coverage guarantee, CV+ does indeed improve over simple CV. For our proofs we introduce a new concept called L\'evy gauge, which can be of independent interest.