Importance Sampling and Necessary Sample Size: an Information Theory Approach

TL;DR

This paper introduces an information theory framework to determine the necessary sample size for importance sampling, relating divergence measures between target and proposal distributions to sampling efficiency.

Contribution

It provides a unified, non-asymptotic approach to derive necessary sample size bounds based on various divergence metrics, extending and complementing existing results.

Findings

Sharper bounds from non-symmetric divergences

Necessary conditions on sample size for importance sampling

Comparison of divergence metrics in sampling efficiency

Abstract

Importance sampling approximates expectations with respect to a target measure by using samples from a proposal measure. The performance of the method over large classes of test functions depends heavily on the closeness between both measures. We derive a general bound that needs to hold for importance sampling to be successful, and relates the $f$-divergence between the target and the proposal to the sample size. The bound is deduced from a new and simple information theory paradigm for the study of importance sampling. As examples of the general theory we give necessary conditions on the sample size in terms of the Kullback-Leibler and $\chi^2$ divergences, and the total variation and Hellinger distances. Our approach is non-asymptotic, and its generality allows to tell apart the relative merits of these metrics. Unsurprisingly, the non-symmetric divergences give sharper bounds than total variation or Hellinger. Our results extend existing necessary conditions -and complement sufficient ones- on the sample size required for importance sampling.