Unbiased estimator for the variance of the leave-one-out cross-validation estimator for a Bayesian normal model with fixed variance

TL;DR

This paper presents a problem-specific unbiased estimator for the variance of leave-one-out cross-validation in a Bayesian normal model, improving uncertainty assessment for model evaluation.

Contribution

It introduces an unbiased variance estimator for LOO-CV tailored to the Bayesian normal model with fixed variance, addressing limitations of general estimators.

Findings

Unbiased estimator for the variance of LOO-CV in a Bayesian normal model.

Demonstrates improved uncertainty quantification for model evaluation.

Shows problem-specific estimators can outperform general approaches.

Abstract

When evaluating and comparing models using leave-one-out cross-validation (LOO-CV), the uncertainty of the estimate is typically assessed using the variance of the sampling distribution. Considering the uncertainty is important, as the variability of the estimate can be high in some cases. An important result by Bengio and Grandvalet (2004) states that no general unbiased variance estimator can be constructed, that would apply for any utility or loss measure and any model. We show that it is possible to construct an unbiased estimator considering a specific predictive performance measure and model. We demonstrate an unbiased sampling distribution variance estimator for the Bayesian normal model with fixed model variance using the expected log pointwise predictive density (elpd) utility score. This example demonstrates that it is possible to obtain improved, problem-specific, unbiased estimators for assessing the uncertainty in LOO-CV estimation.