Evaluating Posterior Distributions by Selectively Breeding Prior Samples

TL;DR

This paper introduces a simple, parallelizable non-MCMC method for approximating posterior distributions by selectively breeding prior samples based on likelihood, avoiding many complexities of traditional MCMC.

Contribution

It proposes a straightforward, tuneless sampling scheme that approximates the posterior by copying prior samples proportionally to their likelihood, with proven convergence properties.

Findings

Method is easy to implement and parallelize.

No rejection, burn-in, or tuning required.

Convergence to the true posterior as sample size increases.

Abstract

Using Markov chain Monte Carlo to sample from posterior distributions was the key innovation which made Bayesian data analysis practical. Notoriously, however, MCMC is hard to tune, hard to diagnose, and hard to parallelize. This pedagogical note explores variants on a universal {\em non}-Markov-chain Monte Carlo scheme for sampling from posterior distributions. The basic idea is to draw parameter values from the prior distributions, evaluate the likelihood of each draw, and then copy that draw a number of times proportional to its likelihood. The distribution after copying is an approximation to the posterior which becomes exact as the number of initial samples goes to infinity; the convergence of the approximation is easily analyzed, and is uniform over Glivenko-Cantelli classes. While not {\em entirely} practical, the schemes are straightforward to implement (a few lines of R), easily parallelized, and require no rejection, burn-in, convergence diagnostics, or tuning of any control settings. I provide references to the prior art which deals with some of the practical obstacles, at some cost in computational and analytical simplicity.