Approximate inference via variational sampling

TL;DR

This paper introduces variational sampling, a novel method for estimating integrals under unnormalized probability distributions, which can outperform importance sampling and other techniques in Bayesian inference tasks.

Contribution

The paper proposes variational sampling, a new approach that fits exponential family models to target distributions using empirical KL divergence minimization, improving approximate inference.

Findings

Faster stochastic convergence than importance sampling under certain conditions

Effective in estimating moments of nearly-Gaussian distributions

Improves Bayesian logistic regression in moderate dimensions

Abstract

A new method called "variational sampling" is proposed to estimate integrals under probability distributions that can be evaluated up to a normalizing constant. The key idea is to fit the target distribution with an exponential family model by minimizing a strongly consistent empirical approximation to the Kullback-Leibler divergence computed using either deterministic or random sampling. It is shown how variational sampling differs conceptually from both quadrature and importance sampling and established that, in the case of random independence sampling, it may have much faster stochastic convergence than importance sampling under mild conditions. The variational sampling implementation presented in this paper requires a rough initial approximation to the target distribution, which may be found, e.g. using the Laplace method, and is shown to then have the potential to substantially improve over several existing approximate inference techniques to estimate moments of order up to two of nearly-Gaussian distributions, which occur frequently in Bayesian analysis. In particular, an application of variational sampling to Bayesian logistic regression in moderate dimension is presented.