Provable Smoothness Guarantees for Black-Box Variational Inference

TL;DR

This paper establishes that for location-scale family approximations in black-box variational inference, the objective inherits Lipschitz smoothness from the target distribution if entropy is excluded, aiding convergence guarantees.

Contribution

It proves that the variational inference objective is Lipschitz smooth under certain conditions, providing theoretical insights and bounds for optimization.

Findings

Objective inherits Lipschitz smoothness from the target distribution.

Provides bounds on the location of optimal parameters.

Offers a key step towards convergence guarantees.

Abstract

Black-box variational inference tries to approximate a complex target distribution though a gradient-based optimization of the parameters of a simpler distribution. Provable convergence guarantees require structural properties of the objective. This paper shows that for location-scale family approximations, if the target is M-Lipschitz smooth, then so is the objective, if the entropy is excluded. The key proof idea is to describe gradients in a certain inner-product space, thus permitting use of Bessel's inequality. This result gives insight into how to parameterize distributions, gives bounds the location of the optimal parameters, and is a key ingredient for convergence guarantees.