Characteristics of Monte Carlo Dropout in Wide Neural Networks

TL;DR

This paper investigates the theoretical properties of Monte Carlo dropout in wide neural networks, showing convergence to Gaussian processes and exploring the effects of finite width and correlation on non-Gaussian behavior.

Contribution

It provides a rigorous analysis of the limiting distribution of wide untrained neural networks under dropout and discusses implications for trained networks and non-Gaussian behaviors.

Findings

Wide untrained NNs under dropout converge to Gaussian processes.

Finite width NNs exhibit non-Gaussian behavior and correlations.

Correlated pre-activations can induce non-Gaussian distributions.

Abstract

Monte Carlo (MC) dropout is one of the state-of-the-art approaches for uncertainty estimation in neural networks (NNs). It has been interpreted as approximately performing Bayesian inference. Based on previous work on the approximation of Gaussian processes by wide and deep neural networks with random weights, we study the limiting distribution of wide untrained NNs under dropout more rigorously and prove that they as well converge to Gaussian processes for fixed sets of weights and biases. We sketch an argument that this property might also hold for infinitely wide feed-forward networks that are trained with (full-batch) gradient descent. The theory is contrasted by an empirical analysis in which we find correlations and non-Gaussian behaviour for the pre-activations of finite width NNs. We therefore investigate how (strongly) correlated pre-activations can induce non-Gaussian behavior in NNs with strongly correlated weights.