Aleatoric and Epistemic Uncertainty Measures for Ordinal Classification through Binary Reduction

TL;DR

This paper introduces novel measures for quantifying aleatoric and epistemic uncertainty in ordinal classification by reducing the problem to binary cases, improving error detection and OOD detection on benchmark datasets.

Contribution

It proposes a new class of uncertainty measures tailored for ordinal classification, addressing a gap in existing methods focused on nominal classification.

Findings

Outperforms standard entropy and variance measures in error detection

Shows competitive performance in out-of-distribution detection

Effectively captures the trade-off between hit-rate and error distance

Abstract

Ordinal classification problems, where labels exhibit a natural order, are prevalent in high-stakes fields such as medicine and finance. Accurate uncertainty quantification, including the decomposition into aleatoric (inherent variability) and epistemic (lack of knowledge) components, is crucial for reliable decision-making. However, existing research has primarily focused on nominal classification and regression. In this paper, we introduce a novel class of measures of aleatoric and epistemic uncertainty in ordinal classification, which is based on a suitable reduction to (entropy- and variance-based) measures for the binary case. These measures effectively capture the trade-off in ordinal classification between exact hit-rate and minimial error distances. We demonstrate the effectiveness of our approach on various tabular ordinal benchmark datasets using ensembles of gradient-boosted trees and multi-layer perceptrons for approximate Bayesian inference. Our method significantly outperforms standard and label-wise entropy and variance-based measures in error detection, as indicated by misclassification rates and mean absolute error. Additionally, the ordinal measures show competitive performance in out-of-distribution (OOD) detection. Our findings highlight the importance of considering the ordinal nature of classification problems when assessing uncertainty.