Unreliable Uncertainty Estimates with Monte Carlo Dropout

TL;DR

This paper critically evaluates Monte Carlo dropout for uncertainty estimation in deep learning, revealing its limitations in accurately capturing true uncertainty compared to Bayesian methods, especially in extrapolation regions.

Contribution

It provides an empirical comparison showing that Monte Carlo dropout often fails to reliably estimate uncertainty, highlighting its limitations relative to Bayesian neural networks and Gaussian processes.

Findings

MCD struggles to reflect true uncertainty accurately.

MCD fails to capture increased uncertainty in extrapolation regions.

Uncertainty estimates from MCD are less reliable than Bayesian approaches.

Abstract

Reliable uncertainty estimation is crucial for machine learning models, especially in safety-critical domains. While exact Bayesian inference offers a principled approach, it is often computationally infeasible for deep neural networks. Monte Carlo dropout (MCD) was proposed as an efficient approximation to Bayesian inference in deep learning by applying neuron dropout at inference time \citep{gal2016dropout}. Hence, the method generates multiple sub-models yielding a distribution of predictions to estimate uncertainty. We empirically investigate its ability to capture true uncertainty and compare to Gaussian Processes (GP) and Bayesian Neural Networks (BNN). We find that MCD struggles to accurately reflect the underlying true uncertainty, particularly failing to capture increased uncertainty in extrapolation and interpolation regions as observed in Bayesian models. The findings suggest that uncertainty estimates from MCD, as implemented and evaluated in these experiments, is not as reliable as those from traditional Bayesian approaches for capturing epistemic and aleatoric uncertainty.