Dropout as a Bayesian Approximation: Appendix

TL;DR

This paper interprets dropout in neural networks as a Bayesian approximation, providing insights into its regularization effects and enabling principled uncertainty estimation in deep learning models.

Contribution

It offers a Bayesian perspective on dropout, explaining its properties and facilitating uncertainty quantification in neural networks.

Findings

Dropout can be viewed as a Bayesian approximation.

This interpretation explains dropout's robustness to overfitting.

Enables principled uncertainty estimation in deep learning.

Abstract

We show that a neural network with arbitrary depth and non-linearities, with dropout applied before every weight layer, is mathematically equivalent to an approximation to a well known Bayesian model. This interpretation might offer an explanation to some of dropout's key properties, such as its robustness to over-fitting. Our interpretation allows us to reason about uncertainty in deep learning, and allows the introduction of the Bayesian machinery into existing deep learning frameworks in a principled way. This document is an appendix for the main paper "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning" by Gal and Ghahramani, 2015.