Generalized Guarantees for Variational Inference in the Presence of Even and Elliptical Symmetry

TL;DR

This paper extends symmetry-based guarantees for variational inference (VI), including broader divergence measures and cases where the target density exhibits partial even and elliptical symmetries, common in hierarchical models.

Contribution

It provides new theoretical guarantees for VI under f-divergences and partial symmetries, broadening understanding of VI's properties in complex models.

Findings

Guarantees for f-divergences including KL and α-divergences.

Strongest guarantees under reverse KL divergence.

Partial symmetry guarantees in hierarchical Bayesian models.

Abstract

We extend several recent results providing symmetry-based guarantees for variational inference (VI) with location-scale families. VI approximates a target density $p$ by the best match $q^*$ in a family $Q$ of tractable distributions that in general does not contain $p$. It is known that VI can recover key properties of $p$, such as its mean and correlation matrix, when $p$ and $Q$ exhibit certain symmetries and $q^*$ is found by minimizing the reverse Kullback-Leibler divergence. We extend these guarantees in two important directions. First, we provide symmetry-based guarantees for $f$-divergences, a broad class that includes the reverse and forward Kullback-Leibler divergences and the $\alpha$-divergences. We highlight properties specific to the reverse Kullback-Leibler divergence under which we obtain our strongest guarantees. Second, we obtain further guarantees for VI when the target density $p$ exhibits even and elliptical symmetries in some but not all of its coordinates. These partial symmetries arise naturally in Bayesian hierarchical models, where the prior induces a challenging geometry but still possesses axes of symmetry. We illustrate these theoretical results in a number of experimental settings.