Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

TL;DR

This paper presents a theoretical framework that interprets dropout in deep neural networks as approximate Bayesian inference, enabling uncertainty estimation without additional computational costs.

Contribution

It introduces a novel perspective linking dropout to Bayesian methods, allowing uncertainty modeling in deep learning models efficiently.

Findings

Dropout can be used to model uncertainty in deep neural networks.

Dropout-based uncertainty improves predictive performance over existing methods.

Application of dropout uncertainty in deep reinforcement learning enhances decision-making.

Abstract

Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computational cost. In this paper we develop a new theoretical framework casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes. A direct result of this theory gives us tools to model uncertainty with dropout NNs -- extracting information from existing models that has been thrown away so far. This mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy. We perform an extensive study of the properties of dropout's uncertainty. Various network architectures and non-linearities are assessed on tasks of regression and classification, using MNIST as an example. We show a considerable improvement in predictive log-likelihood and RMSE compared to existing state-of-the-art methods, and finish by using dropout's uncertainty in deep reinforcement learning.