Fast Second-Order Stochastic Backpropagation for Variational Inference

TL;DR

This paper introduces a second-order stochastic backpropagation method for variational inference that improves convergence rates and scalability, demonstrated on Bayesian logistic regression and VAEs with real-world data.

Contribution

It develops a Hessian-based optimization approach for variational inference using a generalized stochastic backpropagation with a reparametrization trick, enhancing efficiency.

Findings

Significant improvement in convergence rates over existing stochastic gradient methods

Method is practical, scalable, and model-free

Effective on real-world datasets for Bayesian models

Abstract

We propose a second-order (Hessian or Hessian-free) based optimization method for variational inference inspired by Gaussian backpropagation, and argue that quasi-Newton optimization can be developed as well. This is accomplished by generalizing the gradient computation in stochastic backpropagation via a reparametrization trick with lower complexity. As an illustrative example, we apply this approach to the problems of Bayesian logistic regression and variational auto-encoder (VAE). Additionally, we compute bounds on the estimator variance of intractable expectations for the family of Lipschitz continuous function. Our method is practical, scalable and model free. We demonstrate our method on several real-world datasets and provide comparisons with other stochastic gradient methods to show substantial enhancement in convergence rates.