Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison

TL;DR

This paper investigates the properties and uncertainties of Bayesian leave-one-out cross-validation (LOO-CV) for model comparison, highlighting when normal approximations are valid and providing practical guidance for its application.

Contribution

It offers new theoretical and empirical insights into the uncertainty quantification of Bayesian LOO-CV, especially in challenging scenarios like similar models and small data.

Findings

Normal approximation of uncertainty is well calibrated in many cases.

Problems arise with similar models, misspecification, and small datasets.

Skewness in error distribution persists even with large data in certain situations.

Abstract

It is useful to estimate the expected predictive performance of models planned to be used for prediction. We focus on leave-one-out cross-validation (LOO-CV), which has become a popular method for estimating predictive performance of Bayesian models. Given two models, we are interested in comparing the predictive performances and associated uncertainty, which can also be used to compute the probability of one model having better predictive performance than the other model. We study the properties of the Bayesian LOO-CV estimator and the related uncertainty quantification for the predictive performance difference, and analyse when a normal approximation of this uncertainty is well calibrated and whether taking into account higher moments could improve the approximation. We provide new results of the properties both theoretically in the linear regression case and empirically for hierarchical linear, latent linear, and spline models and discuss the challenges. We show that problematic cases include: comparing models with similar predictions, misspecified models, and small data. In these cases, there is a weak connection between the distributions of the LOO-CV estimator and its error. We show that that the problematic skewness of the error distribution for the difference, which occurs when the models make similar predictions, does not fade away when the data size grows to infinity in certain situations. Based on the results, we also provide some practical recommendations for the users of Bayesian LOO-CV for comparing predictive performance of models.