A vanilla Rao--Blackwellization of Metropolis--Hastings algorithms

TL;DR

This paper introduces a universal Rao--Blackwellization scheme for Metropolis--Hastings algorithms that reduces estimator variance with manageable computational costs, supported by theoretical and empirical results.

Contribution

It proposes a new, broadly applicable Rao--Blackwellization method for Metropolis--Hastings algorithms that balances variance reduction and computational efficiency.

Findings

Variance of estimators is significantly reduced.

The scheme guarantees variance reduction in all Metropolis--Hastings estimators.

Performance improvements are demonstrated on toy and real data examples.

Abstract

Casella and Robert [Biometrika 83 (1996) 81--94] presented a general Rao--Blackwellization principle for accept-reject and Metropolis--Hastings schemes that leads to significant decreases in the variance of the resulting estimators, but at a high cost in computation and storage. Adopting a completely different perspective, we introduce instead a universal scheme that guarantees variance reductions in all Metropolis--Hastings-based estimators while keeping the computation cost under control. We establish a central limit theorem for the improved estimators and illustrate their performances on toy examples and on a probit model estimation.