Adaptive Exponential Integration for Stable Gaussian Mixture Black-Box Variational Inference

TL;DR

This paper introduces a stable and efficient Gaussian mixture variational inference method that employs an exponential integrator with adaptive time stepping, ensuring convergence and stability in complex Bayesian problems.

Contribution

It develops a novel framework combining affine-invariant preconditioning, exponential integrators, and adaptive time stepping for stable BBVI with Gaussian mixtures.

Findings

Proves exponential convergence in noise-free settings.

Demonstrates almost-sure convergence with Monte Carlo estimation.

Shows effectiveness on multimodal and PDE-based Bayesian inverse problems.

Abstract

Black-box variational inference (BBVI) with Gaussian mixture families offers a flexible approach for approximating complex posterior distributions without requiring gradients of the target density. However, standard numerical optimization methods often suffer from instability and inefficiency. We develop a stable and efficient framework that combines three key components: (1) affine-invariant preconditioning via natural gradient formulations, (2) an exponential integrator that unconditionally preserves the positive definiteness of covariance matrices, and (3) adaptive time stepping to ensure stability and to accommodate distinct warm-up and convergence phases. The proposed approach has natural connections to manifold optimization and mirror descent. For Gaussian posteriors, we prove exponential convergence in the noise-free setting and almost-sure convergence under Monte Carlo estimation, rigorously justifying the necessity of adaptive time stepping. Numerical experiments on multimodal distributions, Neal's multiscale funnel, and a PDE-based Bayesian inverse problem for Darcy flow demonstrate the effectiveness of the proposed method.