LxCIM: a new rank-based binary classifier performance metric invariant to local exchange of classes

TL;DR

This paper introduces LxCIM, a novel performance metric for binary classifiers that is invariant to local exchange of classes, providing a more robust and interpretable alternative to AUROC, especially in causal discovery contexts.

Contribution

LxCIM is a new rank-based, invariant metric that addresses limitations of AUROC and enhances analysis through the cumulative accuracy-decision rate curve.

Findings

LxCIM is invariant to local exchange of classes.

LxCIM has theoretical links to AUROC, accuracy, and AUDRC.

LxCIM improves causal discovery evaluation.

Abstract

Binary classification is one of the oldest, most prevalent, and studied problems in machine learning. However, the metrics used to evaluate model performance have received comparatively little attention. The area under the receiver operating characteristic curve (AUROC) has long been a standard choice for model comparison. Despite its advantages, AUROC is not always ideal, particularly for problems that are invariant to local exchange of classes (LxC), a new form of metric invariance introduced in this work. To address this limitation, we propose LxCIM (LxC-invariant metric), which is not only rank-based and invariant under local exchange of classes, but also intuitive, logically consistent, and always computable, while enabling more detailed analysis through the cumulative accuracy-decision rate curve. Moreover, LxCIM exhibits clear theoretical connections to AUROC, accuracy, and the area under the accuracy-decision rate curve (AUDRC). These relationships allow for multiple complementary interpretations: as a symmetric form of AUROC, a rank-based analogue of accuracy, or a more representative and more interpretable variant of AUDRC. Finally, we demonstrate the direct applicability of LxCIM to the bivariate causal discovery problem (which exhibits invariance to local exchange of classes) and show how it addresses the acknowledged limitations of existing metrics used in this field. All code and implementation details are publicly available at github.com/tiagobrogueira/Causal-Discovery-In-Exchangeable-Data.