On the Stability of General Bayesian Inference

TL;DR

This paper investigates the stability of Bayesian inference methods, demonstrating that traditional Bayesian updating is fragile to model and data perturbations, while generalized Bayesian methods with the β-divergence offer robust stability in practical scenarios.

Contribution

It shows that generalized Bayesian inference using the β-divergence provides stability across realistic model and data variations, unlike traditional Bayesian methods.

Findings

Traditional Bayesian updating is only stable under strict model assumptions.

Generalized Bayesian inference with β-divergence is stable across practical model and data perturbations.

Stable generalized Bayesian updating retains learning ability in various models.

Abstract

We study the stability of posterior predictive inferences to the specification of the likelihood model and perturbations of the data generating process. In modern big data analyses, useful broad structural judgements may be elicited from the decision-maker but a level of interpolation is required to arrive at a likelihood model. As a result, an often computationally convenient canonical form is used in place of the decision-maker's true beliefs. Equally, in practice, observational datasets often contain unforeseen heterogeneities and recording errors and therefore do not necessarily correspond to how the process was idealised by the decision-maker. Acknowledging such imprecisions, a faithful Bayesian analysis should ideally be stable across reasonable equivalence classes of such inputs. We are able to guarantee that traditional Bayesian updating provides stability across only a very strict class of likelihood models and data generating processes, requiring the decision-maker to elicit their beliefs and understand how the data was generated with an unreasonable degree of accuracy. On the other hand, a generalised Bayesian alternative using the $\beta$-divergence loss function is shown to be stable across practical and interpretable neighbourhoods, providing assurances that posterior inferences are not overly dependent on accidentally introduced spurious specifications or data collection errors. We illustrate this in linear regression, binary classification, and mixture modelling examples, showing that stable updating does not compromise the ability to learn about the data generating process. These stability results provide a compelling justification for using generalised Bayes to facilitate inference under simplified canonical models.