A Unified Evaluation Framework for Epistemic Predictions

TL;DR

This paper introduces a comprehensive evaluation framework for uncertainty-aware classifiers, enabling tailored trade-offs between accuracy and precision across diverse model types and applications.

Contribution

It presents a unified, adaptable evaluation metric applicable to various uncertainty models, facilitating optimal model selection based on specific application needs.

Findings

The framework effectively balances accuracy and precision in different models.

Experimental results on CIFAR-10, MNIST, and CIFAR-100 validate the metric's desired behavior.

Applicable to Bayesian, ensemble, evidential, and other classifiers.

Abstract

Predictions of uncertainty-aware models are diverse, ranging from single point estimates (often averaged over prediction samples) to predictive distributions, to set-valued or credal-set representations. We propose a novel unified evaluation framework for uncertainty-aware classifiers, applicable to a wide range of model classes, which allows users to tailor the trade-off between accuracy and precision of predictions via a suitably designed performance metric. This makes possible the selection of the most suitable model for a particular real-world application as a function of the desired trade-off. Our experiments, concerning Bayesian, ensemble, evidential, deterministic, credal and belief function classifiers on the CIFAR-10, MNIST and CIFAR-100 datasets, show that the metric behaves as desired.