Classifier uncertainty: evidence, potential impact, and probabilistic treatment

TL;DR

This paper introduces a probabilistic method to quantify the uncertainty in classifier performance metrics based on confusion matrices, revealing that many published results may be misleading due to large uncertainties.

Contribution

It provides a simple, classifier-agnostic approach to assess the uncertainty of performance metrics and aids in sample size estimation for desired precision.

Findings

Uncertainties in performance metrics can be surprisingly large.

Many published classifiers' results may be misleading due to uncertainty.

The method is simple, requiring only the confusion matrix.

Abstract

Classifiers are often tested on relatively small data sets, which should lead to uncertain performance metrics. Nevertheless, these metrics are usually taken at face value. We present an approach to quantify the uncertainty of classification performance metrics, based on a probability model of the confusion matrix. Application of our approach to classifiers from the scientific literature and a classification competition shows that uncertainties can be surprisingly large and limit performance evaluation. In fact, some published classifiers are likely to be misleading. The application of our approach is simple and requires only the confusion matrix. It is agnostic of the underlying classifier. Our method can also be used for the estimation of sample sizes that achieve a desired precision of a performance metric.