Variational Message Passing with Structured Inference Networks

TL;DR

This paper introduces a variational message-passing algorithm that integrates structured inference networks into deep probabilistic models, enabling efficient, interpretable, and fast inference by combining structured, amortized, and natural-gradient methods.

Contribution

It proposes structured inference networks for deep models, establishes conditions for fast amortized inference, and derives a variational message passing algorithm for efficient natural-gradient inference.

Findings

Enables structured, amortized, and natural-gradient inference in deep models.

Simplifies and generalizes existing inference methods.

Achieves efficient inference in deep probabilistic graphical models.

Abstract

Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret. We propose a variational message-passing algorithm for variational inference in such models. We make three contributions. First, we propose structured inference networks that incorporate the structure of the graphical model in the inference network of variational auto-encoders (VAE). Second, we establish conditions under which such inference networks enable fast amortized inference similar to VAE. Finally, we derive a variational message passing algorithm to perform efficient natural-gradient inference while retaining the efficiency of the amortized inference. By simultaneously enabling structured, amortized, and natural-gradient inference for deep structured models, our method simplifies and generalizes existing methods.