Uncertainty Quantification with Proper Scoring Rules: Adjusting Measures to Prediction Tasks

TL;DR

This paper introduces a flexible framework for uncertainty quantification using proper scoring rules, allowing adaptation to specific tasks like out-of-distribution detection and active learning, with demonstrated improvements over existing measures.

Contribution

It develops a novel, adaptable approach to quantify uncertainty by decomposing proper scoring rules, enabling task-specific tailoring and improved performance in various applications.

Findings

Mutual information performs best for out-of-distribution detection.

Epistemic uncertainty measure based on zero-one-loss outperforms others in active learning.

Matching scoring rules to task loss enhances uncertainty quantification.

Abstract

We address the problem of uncertainty quantification and propose measures of total, aleatoric, and epistemic uncertainty based on a known decomposition of (strictly) proper scoring rules, a specific type of loss function, into a divergence and an entropy component. This leads to a flexible framework for uncertainty quantification that can be instantiated with different losses (scoring rules), which makes it possible to tailor uncertainty quantification to the use case at hand. We show that this flexibility is indeed advantageous. In particular, we analyze the task of selective prediction and show that the scoring rule should ideally match the task loss. In addition, we perform experiments on two other common tasks. For out-of-distribution detection, our results confirm that a widely used measure of epistemic uncertainty, mutual information, performs best. Moreover, in the setting of active learning, our measure of epistemic uncertainty based on the zero-one-loss consistently outperforms other uncertainty measures.