Statistical comparison of classifiers through Bayesian hierarchical modelling

TL;DR

This paper introduces a Bayesian hierarchical model for comparing classifiers across multiple datasets, providing more reliable probability estimates of their equivalence or difference than traditional null hypothesis tests.

Contribution

It presents a novel Bayesian hierarchical approach that jointly analyzes cross-validation results, reducing estimation error and overcoming limitations of NHST in classifier comparison.

Findings

Reduces estimation error compared to traditional methods

Provides posterior probabilities of classifier equivalence or difference

Improves reliability of classifier comparison across datasets

Abstract

Usually one compares the accuracy of two competing classifiers via null hypothesis significance tests (nhst). Yet the nhst tests suffer from important shortcomings, which can be overcome by switching to Bayesian hypothesis testing. We propose a Bayesian hierarchical model which jointly analyzes the cross-validation results obtained by two classifiers on multiple data sets. It returns the posterior probability of the accuracies of the two classifiers being practically equivalent or significantly different. A further strength of the hierarchical model is that, by jointly analyzing the results obtained on all data sets, it reduces the estimation error compared to the usual approach of averaging the cross-validation results obtained on a given data set.