Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians

TL;DR

This paper provides theoretical insights into variational inference with fixed-variance Gaussian mixtures, framing it as an optimization of mollified relative entropy and analyzing error sources in the process.

Contribution

It introduces a theoretical framework for VI with fixed-variance Gaussian mixtures, linking it to mollified relative entropy and analyzing optimization and approximation errors.

Findings

Gradient descent decreases the mollified relative entropy at each iteration.

The paper bounds the approximation error between finite mixtures and the target distribution.

It characterizes the error sources in variational inference for fixed-variance Gaussian mixtures.

Abstract

Variational inference (VI) is a popular approach in Bayesian inference, that looks for the best approximation of the posterior distribution within a parametric family, minimizing a loss that is typically the (reverse) Kullback-Leibler (KL) divergence. Despite its empirical success, the theoretical properties of VI have only received attention recently, and mostly when the parametric family is the one of Gaussians. This work aims to contribute to the theoretical study of VI in the non-Gaussian case by investigating the setting of Mixture of Gaussians with fixed covariance and constant weights. In this view, VI over this specific family can be casted as the minimization of a Mollified relative entropy, i.e. the KL between the convolution (with respect to a Gaussian kernel) of an atomic measure supported on Diracs, and the target distribution. The support of the atomic measure corresponds to the localization of the Gaussian components. Hence, solving variational inference becomes equivalent to optimizing the positions of the Diracs (the particles), which can be done through gradient descent and takes the form of an interacting particle system. We study two sources of error of variational inference in this context when optimizing the mollified relative entropy. The first one is an optimization result, that is a descent lemma establishing that the algorithm decreases the objective at each iteration. The second one is an approximation error, that upper bounds the objective between an optimal finite mixture and the target distribution.