Cost-Sensitive Evaluation for Binary Classifiers

TL;DR

This paper introduces Weighted Accuracy (WA), a cost-sensitive evaluation metric for binary classifiers that aligns with minimizing total classification cost, clarifies handling class imbalance without rebalancing, and offers a robust estimation procedure.

Contribution

It proposes WA as a straightforward, interpretable metric for cost-sensitive evaluation, providing a conceptual framework for class imbalance management and a method to estimate WA weights.

Findings

WA correlates strongly with Total Classification Cost (TCC) in various scenarios.

Using WA can improve model selection by directly optimizing cost-related performance.

The framework clarifies when rebalancing techniques are beneficial or counterproductive.

Abstract

Selecting an appropriate evaluation metric for classifiers is crucial for model comparison and parameter optimization, yet there is not consensus on a universally accepted metric that serves as a definitive standard. Moreover, there is often a misconception about the perceived need to mitigate imbalance in datasets used to train classification models. Since the final goal in classifier optimization is typically maximizing the return of investment or, equivalently, minimizing the Total Classification Cost (TCC), we define Weighted Accuracy (WA), an evaluation metric for binary classifiers with a straightforward interpretation as a weighted version of the well-known accuracy metric, coherent with the need of minimizing TCC. We clarify the conceptual framework for handling class imbalance in cost-sensitive scenarios, providing an alternative to rebalancing techniques. This framework can be applied to any metric that, like WA, can be expressed as a linear combination of example-dependent quantities and allows for comparing the results obtained in different datasets and for addressing discrepancies between the development dataset, used to train and validate the model, and the target dataset, where the model will be deployed. It also specifies in which scenarios using UCCs-unaware class rebalancing techniques or rebalancing metrics aligns with TCC minimization and when it is instead counterproductive. Finally, we propose a procedure to estimate the WA weight parameter in the absence of fully specified UCCs and demonstrate the robustness of WA by analyzing its correlation with TCC in example-dependent scenarios.