On the Convergence of the ELBO to Entropy Sums

TL;DR

This paper proves that for a broad class of generative models, the variational lower bound at stationary points equals a sum of three entropies, providing a new theoretical insight into the nature of ELBO convergence.

Contribution

It establishes that the ELBO at stationary points equals a sum of entropies for many models, including variational autoencoders, under mild conditions.

Findings

ELBO equals a sum of entropies at stationary points.

The result applies to models with exponential family distributions.

The proof covers finite data points and saddle points.

Abstract

The variational lower bound (a.k.a. ELBO or free energy) is the central objective for many established as well as for many novel algorithms for unsupervised learning. Such algorithms usually increase the bound until parameters have converged to values close to a stationary point of the learning dynamics. Here we show that (for a very large class of generative models) the variational lower bound is at all stationary points of learning equal to a sum of entropies. Concretely, for standard generative models with one set of latents and one set of observed variables, the sum consists of three entropies: (A) the (average) entropy of the variational distributions, (B) the negative entropy of the model's prior distribution, and (C) the (expected) negative entropy of the observable distribution. The obtained result applies under realistic conditions including: finite numbers of data points, at any stationary point (including saddle points) and for any family of (well behaved) variational distributions. The class of generative models for which we show the equality to entropy sums contains many standard as well as novel generative models including standard (Gaussian) variational autoencoders. The prerequisites we use to show equality to entropy sums are relatively mild. Concretely, the distributions defining a given generative model have to be of the exponential family, and the model has to satisfy a parameterization criterion (which is usually fulfilled). Proving equality of the ELBO to entropy sums at stationary points (under the stated conditions) is the main contribution of this work.