Robust performance metrics for imbalanced classification problems

TL;DR

This paper reveals that common binary classification metrics are biased against minority classes in imbalanced settings and proposes robust modifications that maintain meaningful TPR levels.

Contribution

It introduces new robust performance metrics for imbalanced classification that prevent the TPR from approaching zero as class imbalance increases.

Findings

Traditional metrics favor classifiers ignoring the minority class.

Robust metrics keep TPR bounded away from zero in highly imbalanced scenarios.

Numerical experiments demonstrate improved metric behavior on real and simulated data.

Abstract

We show that established performance metrics in binary classification, such as the F-score, the Jaccard similarity coefficient or Matthews' correlation coefficient (MCC), are not robust to class imbalance in the sense that if the proportion of the minority class tends to $0$, the true positive rate (TPR) of the Bayes classifier under these metrics tends to $0$ as well. Thus, in imbalanced classification problems, these metrics favour classifiers which ignore the minority class. To alleviate this issue we introduce robust modifications of the F-score and the MCC for which, even in strongly imbalanced settings, the TPR is bounded away from $0$. We numerically illustrate the behaviour of the various performance metrics in simulations as well as on a credit default data set. We also discuss connections to the ROC and precision-recall curves and give recommendations on how to combine their usage with performance metrics.