Variance Reduction Methods for Dirichlet Expectations

TL;DR

This paper introduces variance reduction techniques for Monte Carlo estimation of Dirichlet expectations, especially effective for large parameters, with applications in topic analysis.

Contribution

It develops importance sampling and control variate methods guided by Laplace approximation to improve estimation accuracy for Dirichlet expectations.

Findings

Importance sampling achieves near-optimal asymptotic error.

Control variates significantly reduce variance.

Methods are demonstrated in topic analysis applications.

Abstract

Dirichlet distributions are probability measures on the unit simplex. They are often used as prior distributions in modeling categorical data, such as in topic analysis of text data. Motivated by this application, we consider Monte Carlo estimation of expectations $\mathbb{E}[\exp(nH(\theta))]$, where $\theta$ has a Dirichlet distribution, $H$ is a real-valued function, and $n$ is a parameter. We develop variance reduction techniques particularly designed to work well for large $n$. Our analysis is guided by the Laplace method for approximating integrals, which we extend to fit our problem setting. We develop an importance sampling method that achieves a near-optimal asymptotic relative error. We use related ideas to select a provably effective control variate. We illustrate these results through their application in topic analysis.