Uncertainty Quantification for Machine Learning: One Size Does Not Fit All

TL;DR

This paper emphasizes that uncertainty quantification in machine learning should be tailored to specific applications, proposing a flexible framework that distinguishes different uncertainty types and aligns measures with task requirements.

Contribution

It introduces a flexible family of uncertainty measures based on proper scoring rules, demonstrating their application to various tasks like selective prediction, out-of-distribution detection, and active learning.

Findings

Mutual information performs best for out-of-distribution detection.

Epistemic uncertainty with zero-one loss outperforms other measures in active learning.

Matching scoring rules to task loss improves selective prediction.

Abstract

Proper quantification of predictive uncertainty is essential for the use of machine learning in safety-critical applications. Various uncertainty measures have been proposed for this purpose, typically claiming superiority over other measures. In this paper, we argue that there is no single best measure. Instead, uncertainty quantification should be tailored to the specific application. To this end, we use a flexible family of uncertainty measures that distinguishes between total, aleatoric, and epistemic uncertainty of second-order distributions. These measures can be instantiated with specific loss functions, so-called proper scoring rules, to control their characteristics, and we show that different characteristics are useful for different tasks. In particular, we show that, for the task of selective prediction, the scoring rule should ideally match the task loss. On the other hand, for out-of-distribution detection, our results confirm that mutual information, a widely used measure of epistemic uncertainty, performs best. Furthermore, in an active learning setting, epistemic uncertainty based on zero-one loss is shown to consistently outperform other uncertainty measures.