Fisher meets Feynman: score-based variational inference with a product of experts

TL;DR

This paper introduces a novel variational inference family using products of $t$-distribution experts, reformulated via a Feynman identity with Dirichlet variables, enabling efficient sampling and optimization.

Contribution

It presents a new BBVI method with a product of experts model reformulated through a Feynman identity, allowing tractable sampling and fast convergence in variational inference.

Findings

Effective modeling of skewed, heavy-tailed, multi-modal distributions.

Convex quadratic program for expert weighting converges exponentially.

Method performs well on synthetic and real-world distributions.

Abstract

We introduce a highly expressive yet distinctly tractable family for black-box variational inference (BBVI). Each member of this family is a weighted product of experts (PoE), and each weighted expert in the product is proportional to a multivariate $t$-distribution. These products of experts can model distributions with skew, heavy tails, and multiple modes, but to use them for BBVI, we must be able to sample from their densities. We show how to do this by reformulating these products of experts as latent variable models with auxiliary Dirichlet random variables. These Dirichlet variables emerge from a Feynman identity, originally developed for loop integrals in quantum field theory, that expresses the product of multiple fractions (or in our case, $t$-distributions) as an integral over the simplex. We leverage this simplicial latent space to draw weighted samples from these products of experts -- samples which BBVI then uses to find the PoE that best approximates a target density. Given a collection of experts, we derive an iterative procedure to optimize the exponents that determine their geometric weighting in the PoE. At each iteration, this procedure minimizes a regularized Fisher divergence to match the scores of the variational and target densities at a batch of samples drawn from the current approximation. This minimization reduces to a convex quadratic program, and we prove under general conditions that these updates converge exponentially fast to a near-optimal weighting of experts. We conclude by evaluating this approach on a variety of synthetic and real-world target distributions.