On importance sampling and independent Metropolis-Hastings with an unbounded weight function

TL;DR

This paper analyzes the approximation errors and bias properties of importance sampling and independent Metropolis-Hastings algorithms when the weight function is unbounded but has finite moments, providing bounds, bias comparisons, and unbiased estimator techniques.

Contribution

It introduces bounds on the total variation distance for PIMH with unbounded weights, compares biases of importance sampling and IMH, and proposes unbiased estimators with finite moments.

Findings

IMH has smaller finite-time bias than importance sampling.

Maximal coupling is used to derive bounds on chain convergence.

Unbiased estimators with finite moments are constructed under certain conditions.

Abstract

Importance sampling and independent Metropolis-Hastings (IMH) are among the fundamental building blocks of Monte Carlo methods. Both require a proposal distribution that globally approximates the target distribution. The Radon-Nikodym derivative of the target distribution relative to the proposal is called the weight function. Under the assumption that the weight is unbounded but has finite moments under the proposal distribution, we study the approximation error of importance sampling and of the particle independent Metropolis-Hastings algorithm (PIMH), which includes IMH as a special case. For the chains generated by such algorithms, we show that the common random numbers coupling is maximal. Using that coupling we derive bounds on the total variation distance of a PIMH chain to its target distribution. Our results allow a formal comparison of the finite-time biases of importance sampling and IMH, and we find the latter to be have a smaller bias. We further consider bias removal techniques using couplings, and provide conditions under which the resulting unbiased estimators have finite moments. These unbiased estimators provide an alternative to self-normalized importance sampling, implementable in the same settings. We compare their asymptotic efficiency as the number of particles goes to infinity, and consider their use in robust mean estimation techniques.