Second-Order Uncertainty Quantification: Variance-Based Measures

TL;DR

This paper introduces a new variance-based method for quantifying uncertainty in classification models, focusing on second-order distributions to enable class-level reasoning and improve decision-making.

Contribution

It proposes a novel variance-based uncertainty measure that captures second-order distribution properties and satisfies key axiomatic properties, offering an alternative to entropy-based measures.

Findings

Measures are effective in uncertainty quantification.

They are competitive with entropy-based measures.

The approach enables class-based uncertainty reasoning.

Abstract

Uncertainty quantification is a critical aspect of machine learning models, providing important insights into the reliability of predictions and aiding the decision-making process in real-world applications. This paper proposes a novel way to use variance-based measures to quantify uncertainty on the basis of second-order distributions in classification problems. A distinctive feature of the measures is the ability to reason about uncertainties on a class-based level, which is useful in situations where nuanced decision-making is required. Recalling some properties from the literature, we highlight that the variance-based measures satisfy important (axiomatic) properties. In addition to this axiomatic approach, we present empirical results showing the measures to be effective and competitive to commonly used entropy-based measures.