Posterior predictive p-values and the convex order

TL;DR

This paper analyzes the distribution and properties of posterior predictive p-values in Bayesian model checking, revealing their variability, potential non-conservativeness, and methods for correction to ensure reliable inference.

Contribution

It provides a theoretical characterization of posterior predictive p-values using convex order, and proposes correction methods to improve their conservativeness.

Findings

Posterior predictive p-values are exactly those less variable than uniform on [0,1] in convex order.

They can be non-conservative, leading to higher false rejection rates.

Correction methods can restore conservatism in various practical scenarios.

Abstract

Posterior predictive p-values are a common approach to Bayesian model-checking. This article analyses their frequency behaviour, that is, their distribution when the parameters and the data are drawn from the prior and the model respectively. We show that the family of possible distributions is exactly described as the distributions that are less variable than uniform on [0,1], in the convex order. In general, p-values with such a property are not conservative, and we illustrate how the theoretical worst-case error rate for false rejection can occur in practice. We describe how to correct the p-values to recover conservatism in several common scenarios, for example, when interpreting a single p-value or when combining multiple p-values into an overall score of significance. We also handle the case where the p-value is estimated from posterior samples obtained from techniques such as Markov Chain or Sequential Monte Carlo. Our results place posterior predictive p-values in a much clearer theoretical framework, allowing them to be used with more assurance.