Sandwiching the marginal likelihood using bidirectional Monte Carlo

TL;DR

The paper introduces bidirectional Monte Carlo, a method that provides accurate bounds on the marginal likelihood by combining forward and reverse stochastic algorithms, enabling better evaluation of ML estimators.

Contribution

It presents a novel bidirectional Monte Carlo technique that accurately bounds the marginal likelihood, improving estimation and evaluation of ML in complex models.

Findings

Provides high-probability stochastic bounds on log-ML

Evaluates existing ML estimators across multiple models

Offers insights into accurate marginal likelihood estimation

Abstract

Computing the marginal likelihood (ML) of a model requires marginalizing out all of the parameters and latent variables, a difficult high-dimensional summation or integration problem. To make matters worse, it is often hard to measure the accuracy of one's ML estimates. We present bidirectional Monte Carlo, a technique for obtaining accurate log-ML estimates on data simulated from a model. This method obtains stochastic lower bounds on the log-ML using annealed importance sampling or sequential Monte Carlo, and obtains stochastic upper bounds by running these same algorithms in reverse starting from an exact posterior sample. The true value can be sandwiched between these two stochastic bounds with high probability. Using the ground truth log-ML estimates obtained from our method, we quantitatively evaluate a wide variety of existing ML estimators on several latent variable models: clustering, a low rank approximation, and a binary attributes model. These experiments yield insights into how to accurately estimate marginal likelihoods.