Fearless Stochasticity in Expectation Propagation

TL;DR

This paper introduces two novel variants of expectation propagation that are more stable, sample-efficient, and easier to tune, especially when using Monte Carlo estimates, by leveraging a natural-gradient perspective.

Contribution

The paper provides a new interpretation of EP updates as natural-gradient optimization and proposes two variants optimized for Monte Carlo estimation.

Findings

New EP variants are stable with single-sample MC estimates.

Proposed methods improve speed-accuracy trade-off.

Variants are easier to tune and do not need debiasing estimators.

Abstract

Expectation propagation (EP) is a family of algorithms for performing approximate inference in probabilistic models. The updates of EP involve the evaluation of moments -- expectations of certain functions -- which can be estimated from Monte Carlo (MC) samples. However, the updates are not robust to MC noise when performed naively, and various prior works have attempted to address this issue in different ways. In this work, we provide a novel perspective on the moment-matching updates of EP; namely, that they perform natural-gradient-based optimisation of a variational objective. We use this insight to motivate two new EP variants, with updates that are particularly well-suited to MC estimation. They remain stable and are most sample-efficient when estimated with just a single sample. These new variants combine the benefits of their predecessors and address key weaknesses. In particular, they are easier to tune, offer an improved speed-accuracy trade-off, and do not rely on the use of debiasing estimators. We demonstrate their efficacy on a variety of probabilistic inference tasks.