Concentration and Calibration in Predictive Bayesian Inference

TL;DR

This paper analyzes predictive Bayesian inference (PBI), showing that its reliability heavily depends on the accuracy of the forward predictive model used, which affects uncertainty quantification and calibration.

Contribution

It demonstrates that PBI posterior concentration depends on the predictive model and highlights the importance of model accuracy for calibrated inferences.

Findings

Posterior concentrates on a quantity depending on the predictive model.

Uncertainty quantification in PBI is fully determined by the forward predictive model.

Inaccuracy in the predictive model can lead to arbitrarily poor coverage of credible sets.

Abstract

Predictive Bayesian inference (PBI) represents a model-and prior-agnostic approach to standard Bayesian inference which allows users to quantify uncertainty for a functional of interest only by specifying a forward predictive model for future unobserved data. The flexibility and generality of this framework have led to a host of novel algorithms for implementing this approach, and many empirical applications, yet the reliability of the resulting inferences for the underlying statistical functional of interest remains unclear. Herein, we demonstrate that when using PBI for a population functional of interest, the resulting posterior concentrates onto a well-defined quantity that explicitly depends on the forward predictive model used to implement the predictive recursion underlying the method. Furthermore, the forward predictive model entirely determines the uncertainty quantification produced in PBI. Consequently, our results show that if the predictive model does not capture all relevant features of the data, and, even in very simple examples, the coverage of predictive Bayes credible sets for the population value of the functional of interest can be arbitrarily close to zero. We carefully explain why this occurs, and show that this behavior is directly tied to the inaccuracy of the forward predictive model used to produce future observations within the PBI framework. As a consequence, our results imply that in order for PBI to deliver calibrated posterior inferences, the resulting predictive engine used to generate posterior samples must contain, in a well-defined sense, the true DGP, else inferences generated under this framework will not be calibrated.