Uncertainty Quantification for Deep Learning

TL;DR

This paper reviews current methods for uncertainty quantification in deep learning, identifies inconsistencies, and proposes a comprehensive, statistically consistent framework that improves model robustness and reliability in real-world applications.

Contribution

It introduces a unified framework for uncertainty quantification in deep learning that accounts for all major sources of uncertainty using Bayesian methods and provides practical implementation guidance.

Findings

Uncertainty from training data dominates in real-world predictions.

Explicit modeling of training data uncertainty improves robustness.

The framework enhances model reliability in practical scenarios.

Abstract

We present a critical survey on the consistency of uncertainty quantification used in deep learning and highlight partial uncertainty coverage and many inconsistencies. We then provide a comprehensive and statistically consistent framework for uncertainty quantification in deep learning that accounts for all major sources of uncertainty: input data, training and testing data, neural network weights, and machine-learning model imperfections, targeting regression problems. We systematically quantify each source by applying Bayes' theorem and conditional probability densities and introduce a fast, practical implementation method. We demonstrate its effectiveness on a simple regression problem and a real-world application: predicting cloud autoconversion rates using a neural network trained on aircraft measurements from the Azores and guided by a two-moment bin model of the stochastic collection equation. In this application, uncertainty from the training and testing data dominates, followed by input data, neural network model, and weight variability. Finally, we highlight the practical advantages of this methodology, showing that explicitly modeling training data uncertainty improves robustness to new inputs that fall outside the training data, and enhances model reliability in real-world scenarios.