On numerical approximation schemes for expectation propagation

TL;DR

This paper evaluates various numerical approximation methods for expectation propagation in large-scale learning, finding variational sampling most effective for convergence, while Gaussian quadrature often underperforms.

Contribution

It compares multiple approximation strategies for expectation propagation, highlighting the effectiveness of variational sampling in large-scale classification tasks.

Findings

Variational sampling leads to the best convergence in expectation propagation.

Laplace-style methods work well with smooth factors but are unstable with non-differentiable ones.

Gaussian quadrature often results in unstable behavior or sub-optimal solutions.

Abstract

Several numerical approximation strategies for the expectation-propagation algorithm are studied in the context of large-scale learning: the Laplace method, a faster variant of it, Gaussian quadrature, and a deterministic version of variational sampling (i.e., combining quadrature with variational approximation). Experiments in training linear binary classifiers show that the expectation-propagation algorithm converges best using variational sampling, while it also converges well using Laplace-style methods with smooth factors but tends to be unstable with non-differentiable ones. Gaussian quadrature yields unstable behavior or convergence to a sub-optimal solution in most experiments.