Classifier comparison using precision

TL;DR

This paper surveys statistical methods for comparing classifiers based on precision, addressing inter-precision correlation, and extending to multi-class and cross-validation scenarios, with a partial Bayesian approach for known class prevalence.

Contribution

It introduces a comprehensive review of statistical tests for precision-based classifier comparison, including a Bayesian update, which is novel in this context.

Findings

Methods for global null hypothesis testing of precision differences.

Extensions to multi-class classifiers and cross-validation.

Application to deep architecture comparisons.

Abstract

New proposed models are often compared to state-of-the-art using statistical significance testing. Literature is scarce for classifier comparison using metrics other than accuracy. We present a survey of statistical methods that can be used for classifier comparison using precision, accounting for inter-precision correlation arising from use of same dataset. Comparisons are made using per-class precision and methods presented to test global null hypothesis of an overall model comparison. Comparisons are extended to multiple multi-class classifiers and to models using cross validation or its variants. Partial Bayesian update to precision is introduced when population prevalence of a class is known. Applications to compare deep architectures are studied.