Approximate Inference with the Variational Holder Bound

TL;DR

The paper proposes the Variational Holder bound as a convex alternative to Variational Bayes for approximate Bayesian inference, enabling the use of convex optimization techniques and providing new analysis tools.

Contribution

It introduces the VH bound, a convex upper bound for Bayesian inference, allowing convex optimization methods to be used instead of non-convex variational approaches.

Findings

VH bound is convex and can be optimized with standard convex algorithms.

Experiments show VH performs well compared to VB, EP, and numerical methods.

The approximation error of VH can be analyzed using convex optimization tools.

Abstract

We introduce the Variational Holder (VH) bound as an alternative to Variational Bayes (VB) for approximate Bayesian inference. Unlike VB which typically involves maximization of a non-convex lower bound with respect to the variational parameters, the VH bound involves minimization of a convex upper bound to the intractable integral with respect to the variational parameters. Minimization of the VH bound is a convex optimization problem; hence the VH method can be applied using off-the-shelf convex optimization algorithms and the approximation error of the VH bound can also be analyzed using tools from convex optimization literature. We present experiments on the task of integrating a truncated multivariate Gaussian distribution and compare our method to VB, EP and a state-of-the-art numerical integration method for this problem.