Bayesian Computation in Deep Learning

TL;DR

This paper introduces Bayesian computation techniques like SG-MCMC and variational inference for deep learning, emphasizing their role in improving model uncertainty estimation in neural networks and generative models.

Contribution

It provides an overview of approximate Bayesian inference methods applied to deep learning, focusing on their challenges and solutions in neural networks and generative models.

Findings

Review of stochastic gradient MCMC and variational inference methods

Discussion of challenges in Bayesian posterior inference for deep models

Insights into solutions for scalable Bayesian computation in deep learning

Abstract

Bayesian methods have shown success in deep learning applications. For example, in predictive tasks, Bayesian neural networks leverage Bayesian reasoning of model uncertainty to improve the reliability and uncertainty awareness of deep neural networks. In generative modeling domain, many widely used deep generative models, such as deep latent variable models, require approximate Bayesian inference to infer their latent variables for the training. In this chapter, we provide an introduction to approximate inference techniques as Bayesian computation methods applied to deep learning models, with a focus on Bayesian neural networks and deep generative models. We review two arguably most popular approximate Bayesian computational methods, stochastic gradient Markov chain Monte Carlo (SG-MCMC) and variational inference (VI), and explain their unique challenges in posterior inference as well as the solutions when applied to deep learning models.