Optimal Hold-Out Size in Cross-Validation

TL;DR

This paper develops a principled method for selecting the optimal hold-out size in cross-validation by balancing training accuracy and evaluation uncertainty, improving model comparison and scientific inference.

Contribution

It introduces a utility-based rule for choosing the hold-out size based on irreducible noise, formalizing the tradeoff and providing data-dependent, assumption-aware guidance.

Findings

Optimal K depends on data and model complexity.

Choice of K varies with irreducible error assumptions.

Impacts on scientific conclusions and model comparisons.

Abstract

Cross-validation (CV) is routinely used across the sciences to select models and tune parameters, and the resulting choices are often interpreted as substantive scientific conclusions (e.g., which variables, mechanisms, or risk factors are ``supported by the data''). A key part of the CV procedure -- the hold-out size, or equivalently the fold count $K$ -- is typically set by convention (e.g., 80/20, $K=5$) rather than by a principled criterion. Central to the issue is the tradeoff between training and testing: increasing the training sample size improves model accuracy, while sacrificing certainty around the accuracy itself. We formalize the tradeoff by targeting predictive performance and explicitly penalizing evaluation uncertainty, which cannot be identified from the data without additional assumptions. We derive finite-sample expressions of this evaluation uncertainty under symmetric errors and general upper bounds under broader error conditions, yielding a transparent utility-based rule for selecting the hold-out size as a function of an irreducible-noise parameter. Empirical analyses with linear regression and random forests across multiple domains, and a high-dimensional genomics application, show that (i) the choice of $K$ is dependent on the data and model. (ii) the optimal $K$ varies based on the assumption on the irreducible error, and (iii) the implied inferential conclusions can change materially as the irreducible error, and thus $K$, varies. The resulting framework replaces a one-size-fits-all convention with a context-specific, assumption-explicit choice of $K$, enabling more reliable model comparisons and downstream scientific inference.