Independent projections of diffusions: Gradient flows for variational inference and optimal mean field approximations

TL;DR

This paper introduces the independent projection, an optimal method for approximating high-dimensional diffusions with independent coordinates, connecting gradient flows, variational inference, and convergence analysis.

Contribution

It proposes the independent projection as an optimal approximation method, linking it to Wasserstein gradient flows and providing convergence results, including in the log-concave case.

Findings

Independent projection is the Wasserstein gradient flow for relative entropy constrained to product measures.

Provides qualitative and quantitative convergence results, especially for log-concave measures.

Among processes with independent coordinates, it has the slowest path-space entropy growth.

Abstract

What is the optimal way to approximate a high-dimensional diffusion process by one in which the coordinates are independent? This paper presents a construction, called the \emph{independent projection}, which is optimal for two natural criteria. First, when the original diffusion is reversible with invariant measure $\rho_*$, the independent projection serves as the Wasserstein gradient flow for the relative entropy $H(\cdot\,|\,\rho_*)$ constrained to the space of product measures. This is related to recent Langevin-based sampling schemes proposed in the statistical literature on mean field variational inference. In addition, we provide both qualitative and quantitative results on the long-time convergence of the independent projection, with quantitative results in the log-concave case derived via a new variant of the logarithmic Sobolev inequality. Second, among all processes with independent coordinates, the independent projection is shown to exhibit the slowest growth rate of path-space entropy relative to the original diffusion. This sheds new light on the classical McKean-Vlasov equation and recent variants proposed for non-exchangeable systems, which can be viewed as special cases of the independent projection.