The FMRIB Variational Bayesian Inference Tutorial II: Stochastic Variational Bayes

TL;DR

This paper introduces a stochastic variational Bayesian inference method that improves upon traditional approaches by reducing limitations and leveraging computational techniques from machine learning, enhancing Bayesian parameter estimation.

Contribution

It presents a novel stochastic approach to Variational Bayes that overcomes some limitations of earlier mean field methods, inspired by machine learning algorithms.

Findings

The stochastic VB method offers more flexible posterior approximations.

It reduces computational constraints of traditional VB methods.

The approach aligns Bayesian inference with modern machine learning techniques.

Abstract

Bayesian methods have proved powerful in many applications for the inference of model parameters from data. These methods are based on Bayes' theorem, which itself is deceptively simple. However, in practice the computations required are intractable even for simple cases. Hence methods for Bayesian inference have historically either been significantly approximate, e.g., the Laplace approximation, or achieve samples from the exact solution at significant computational expense, e.g., Markov Chain Monte Carlo methods. Since around the year 2000 so-called Variational approaches to Bayesian inference have been increasingly deployed. In its most general form Variational Bayes (VB) involves approximating the true posterior probability distribution via another more 'manageable' distribution, the aim being to achieve as good an approximation as possible. In the original FMRIB Variational Bayes tutorial we documented an approach to VB based that took a 'mean field' approach to forming the approximate posterior, required the conjugacy of prior and likelihood, and exploited the Calculus of Variations, to derive an iterative series of update equations, akin to Expectation Maximisation. In this tutorial we revisit VB, but now take a stochastic approach to the problem that potentially circumvents some of the limitations imposed by the earlier methodology. This new approach bears a lot of similarity to, and has benefited from, computational methods applied to machine learning algorithms. Although, what we document here is still recognisably Bayesian inference in the classic sense, and not an attempt to use machine learning as a black-box to solve the inference problem.