Quantifying Epistemic Uncertainty in Deep Learning

TL;DR

This paper introduces a theoretical framework to dissect and estimate epistemic uncertainty in deep learning, distinguishing procedural and data variability, and proposes practical methods to quantify these uncertainties effectively.

Contribution

It provides the first comprehensive framework to separate and estimate epistemic uncertainty components in deep learning, with practical influence function and batching methods.

Findings

Framework successfully dissects uncertainty into procedural and data components.

Proposed methods overcome computational challenges of classical statistical techniques.

Experimental results validate the theoretical framework and demonstrate practical utility.

Abstract

Uncertainty quantification is at the core of the reliability and robustness of machine learning. In this paper, we provide a theoretical framework to dissect the uncertainty, especially the \textit{epistemic} component, in deep learning into \textit{procedural variability} (from the training procedure) and \textit{data variability} (from the training data), which is the first such attempt in the literature to our best knowledge. We then propose two approaches to estimate these uncertainties, one based on influence function and one on batching. We demonstrate how our approaches overcome the computational difficulties in applying classical statistical methods. Experimental evaluations on multiple problem settings corroborate our theory and illustrate how our framework and estimation can provide direct guidance on modeling and data collection efforts.