R\'enyi Divergence Variational Inference

TL;DR

This paper proposes the variational Rényi bound (VR), a flexible framework for variational inference that interpolates between different divergence measures, enabling improved inference in Bayesian neural networks and auto-encoders.

Contribution

It introduces the VR bound extending variational inference to Rényi's alpha-divergences, unifying existing methods and allowing smooth interpolation controlled by alpha.

Findings

Demonstrates applicability to Bayesian neural networks

Shows effectiveness in variational auto-encoders

Provides a tractable optimization framework

Abstract

This paper introduces the variational R\'enyi bound (VR) that extends traditional variational inference to R\'enyi's alpha-divergences. This new family of variational methods unifies a number of existing approaches, and enables a smooth interpolation from the evidence lower-bound to the log (marginal) likelihood that is controlled by the value of alpha that parametrises the divergence. The reparameterization trick, Monte Carlo approximation and stochastic optimisation methods are deployed to obtain a tractable and unified framework for optimisation. We further consider negative alpha values and propose a novel variational inference method as a new special case in the proposed framework. Experiments on Bayesian neural networks and variational auto-encoders demonstrate the wide applicability of the VR bound.