LOO-PIT predictive model checking

TL;DR

This paper introduces LOO-PIT, a Bayesian model checking method that accounts for dependencies in predictive distributions, with new tests and visualization tools demonstrating improved detection of model miscalibration.

Contribution

It develops dependency-aware tests and visualization techniques for LOO-PIT, enhancing Bayesian model assessment accuracy over traditional methods.

Findings

Proves dependency in LOO-PIT values is significant and model-dependent.

Proposes three tests and a graphical method for detecting departures from uniformity.

Demonstrates higher power and better performance of the proposed methods in experiments.

Abstract

We consider predictive checking for Bayesian model assessment using leave-one-out probability integral transform (LOO-PIT). LOO-PIT values are conditional cumulative predictive probabilities given LOO predictive distributions and corresponding left out observations. For a well-calibrated model, LOO-PIT values should be near uniformly distributed, but in the finite sample case they are not independent, due to LOO predictive distributions being determined by nearly the same data (all but one observation). We prove that this dependency is non-negligible in the finite case and depends on model complexity. We propose three testing procedures that can be used for continuous and discrete dependent uniform values. We also propose an automated graphical method for visualizing local departures from the null. Extensive numerical experiments on simulated and real datasets demonstrate that the proposed tests achieve competitive performance overall and have much higher power than standard uniformity tests based on the independence assumption that inevitably lead to lower than expected rejection rate.