Global consensus Monte Carlo

TL;DR

This paper introduces a distributed MCMC and SMC framework for Bayesian inference on large datasets, using hierarchical models with auxiliary parameters to balance computational efficiency and accuracy.

Contribution

It proposes a novel hierarchical model with auxiliary parameters inspired by consensus optimization, enabling distributed Bayesian inference with controlled approximation quality.

Findings

Effective distributed MCMC algorithm demonstrated on simulated data.

SMC sampler with adaptive association strengths improves inference accuracy.

Few distributional assumptions required compared to similar methods.

Abstract

To conduct Bayesian inference with large data sets, it is often convenient or necessary to distribute the data across multiple machines. We consider a likelihood function expressed as a product of terms, each associated with a subset of the data. Inspired by global variable consensus optimisation, we introduce an instrumental hierarchical model associating auxiliary statistical parameters with each term, which are conditionally independent given the top-level parameters. One of these top-level parameters controls the unconditional strength of association between the auxiliary parameters. This model leads to a distributed MCMC algorithm on an extended state space yielding approximations of posterior expectations. A trade-off between computational tractability and fidelity to the original model can be controlled by changing the association strength in the instrumental model. We further propose the use of a SMC sampler with a sequence of association strengths, allowing both the automatic determination of appropriate strengths and for a bias correction technique to be applied. In contrast to similar distributed Monte Carlo algorithms, this approach requires few distributional assumptions. The performance of the algorithms is illustrated with a number of simulated examples.