Foundations of the Theory of Performance-Based Ranking

TL;DR

This paper develops a universal, axiomatic framework for performance-based ranking of entities, integrating probability and order theories, and demonstrates its applicability to classification metrics.

Contribution

It introduces the first axiomatic definition of performance orderings and a universal family of ranking scores considering application-specific preferences.

Findings

Encompasses well-known classification scores like accuracy, recall, and precision.

Identifies scores unsuitable for performance orderings under the axioms.

Provides a rigorous mathematical foundation for performance ranking methods.

Abstract

Ranking entities such as algorithms, devices, methods, or models based on their performances, while accounting for application-specific preferences, is a challenge. To address this challenge, we establish the foundations of a universal theory for performance-based ranking. First, we introduce a rigorous framework built on top of both the probability and order theories. Our new framework encompasses the elements necessary to (1) manipulate performances as mathematical objects, (2) express which performances are worse than or equivalent to others, (3) model tasks through a variable called satisfaction, (4) consider properties of the evaluation, (5) define scores, and (6) specify application-specific preferences through a variable called importance. On top of this framework, we propose the first axiomatic definition of performance orderings and performance-based rankings. Then, we introduce a universal parametric family of scores, called ranking scores, that can be used to establish rankings satisfying our axioms, while considering application-specific preferences. Finally, we show, in the case of two-class classification, that the family of ranking scores encompasses well-known performance scores, including the accuracy, the true positive rate (recall, sensitivity), the true negative rate (specificity), the positive predictive value (precision), and F1. However, we also show that some other scores commonly used to compare classifiers are unsuitable to derive performance orderings satisfying the axioms.