Expectation Propagation in the large-data limit

TL;DR

This paper analyzes Expectation Propagation (EP) in the large-data limit, showing it behaves like Newton's method and is asymptotically exact, providing new theoretical insights into its convergence and divergence properties.

Contribution

Introduces a variant of EP called averaged-EP and proves that EP behaves like Newton's method in the large-data limit, establishing its asymptotic exactness.

Findings

EP behaves like Newton's method for finding the mode

EP is asymptotically exact in the large-data limit

Poor initialization can cause EP to diverge

Abstract

Expectation Propagation (Minka, 2001) is a widely successful algorithm for variational inference. EP is an iterative algorithm used to approximate complicated distributions, typically to find a Gaussian approximation of posterior distributions. In many applications of this type, EP performs extremely well. Surprisingly, despite its widespread use, there are very few theoretical guarantees on Gaussian EP, and it is quite poorly understood. In order to analyze EP, we first introduce a variant of EP: averaged-EP (aEP), which operates on a smaller parameter space. We then consider aEP and EP in the limit of infinite data, where the overall contribution of each likelihood term is small and where posteriors are almost Gaussian. In this limit, we prove that the iterations of both aEP and EP are simple: they behave like iterations of Newton's algorithm for finding the mode of a function. We use this limit behavior to prove that EP is asymptotically exact, and to obtain other insights into the dynamic behavior of EP: for example, that it may diverge under poor initialization exactly like Newton's method. EP is a simple algorithm to state, but a difficult one to study. Our results should facilitate further research into the theoretical properties of this important method.