Stochastic Variational Inference via Upper Bound

TL;DR

This paper introduces a new, tighter upper bound for evidence in stochastic variational inference, improving scalability and performance in Bayesian deep learning applications.

Contribution

The paper proposes the Evidence Upper Bound (EUBO), a novel surrogate loss that is tighter than previous bounds, and develops an SGD-based optimization method for scalable inference.

Findings

EUBO is tighter than previous divergence-based bounds.

EUBO-VI outperforms state-of-the-art methods in Bayesian neural networks.

The bounds effectively sandwich the evidence in simulations.

Abstract

Stochastic variational inference (SVI) plays a key role in Bayesian deep learning. Recently various divergences have been proposed to design the surrogate loss for variational inference. We present a simple upper bound of the evidence as the surrogate loss. This evidence upper bound (EUBO) equals to the log marginal likelihood plus the KL-divergence between the posterior and the proposal. We show that the proposed EUBO is tighter than previous upper bounds introduced by $\chi$-divergence or $\alpha$-divergence. To facilitate scalable inference, we present the numerical approximation of the gradient of the EUBO and apply the SGD algorithm to optimize the variational parameters iteratively. Simulation study with Bayesian logistic regression shows that the upper and lower bounds well sandwich the evidence and the proposed upper bound is favorably tight. For Bayesian neural network, the proposed EUBO-VI algorithm outperforms state-of-the-art results for various examples.