Measuring Classification Decision Certainty and Doubt

TL;DR

This paper introduces intuitive certainty and doubt scores to quantify and compare the uncertainty of classification predictions within Bayesian and frequentist frameworks.

Contribution

It proposes novel scores for assessing prediction uncertainty in classification tasks applicable to both Bayesian and frequentist approaches.

Findings

Certainty and doubt scores effectively quantify prediction confidence.

Scores enable comparison of uncertainty across different models.

Applicable to multi-class classification problems.

Abstract

Quantitative characterizations and estimations of uncertainty are of fundamental importance in optimization and decision-making processes. Herein, we propose intuitive scores, which we call certainty and doubt, that can be used in both a Bayesian and frequentist framework to assess and compare the quality and uncertainty of predictions in (multi-)classification decision machine learning problems.