Stochastic Variational Inference with Gradient Linearization

TL;DR

This paper introduces SVIGL, a stochastic variational inference method that uses gradient linearization to improve convergence speed and approximation quality, demonstrated across three diverse applications.

Contribution

The paper proposes SVIGL, a novel stochastic variational inference technique utilizing gradient linearization for faster convergence and better approximations.

Findings

SVIGL converges faster than standard stochastic variational inference.

SVIGL achieves comparable or better KL divergence in approximations.

SVIGL improves results in optical flow, denoising, and surface reconstruction.

Abstract

Variational inference has experienced a recent surge in popularity owing to stochastic approaches, which have yielded practical tools for a wide range of model classes. A key benefit is that stochastic variational inference obviates the tedious process of deriving analytical expressions for closed-form variable updates. Instead, one simply needs to derive the gradient of the log-posterior, which is often much easier. Yet for certain model classes, the log-posterior itself is difficult to optimize using standard gradient techniques. One such example are random field models, where optimization based on gradient linearization has proven popular, since it speeds up convergence significantly and can avoid poor local optima. In this paper we propose stochastic variational inference with gradient linearization (SVIGL). It is similarly convenient as standard stochastic variational inference - all that is required is a local linearization of the energy gradient. Its benefit over stochastic variational inference with conventional gradient methods is a clear improvement in convergence speed, while yielding comparable or even better variational approximations in terms of KL divergence. We demonstrate the benefits of SVIGL in three applications: Optical flow estimation, Poisson-Gaussian denoising, and 3D surface reconstruction.