Parallelizing MCMC via Weierstrass Sampler

TL;DR

This paper introduces the Weierstrass sampler, a parallel MCMC method that efficiently combines independent subset posteriors, improving computational speed for large-scale Bayesian analysis while maintaining controlled approximation error.

Contribution

The paper proposes a novel Weierstrass sampler for parallel MCMC that approximates full data posteriors by combining subset chains, offering higher efficiency and bounded approximation error.

Findings

The Weierstrass sampler is competitive with existing methods.

Approximation error is controllable via tuning parameters.

Simulation studies demonstrate improved computational efficiency.

Abstract

With the rapidly growing scales of statistical problems, subset based communication-free parallel MCMC methods are a promising future for large scale Bayesian analysis. In this article, we propose a new Weierstrass sampler for parallel MCMC based on independent subsets. The new sampler approximates the full data posterior samples via combining the posterior draws from independent subset MCMC chains, and thus enjoys a higher computational efficiency. We show that the approximation error for the Weierstrass sampler is bounded by some tuning parameters and provide suggestions for choice of the values. Simulation study shows the Weierstrass sampler is very competitive compared to other methods for combining MCMC chains generated for subsets, including averaging and kernel smoothing.