Measuring Class-Imbalance Sensitivity of Deterministic Performance Evaluation Metrics

TL;DR

This paper introduces a framework to quantify how sensitive different evaluation metrics are to class imbalance, revealing a logarithmic relationship that helps interpret metric reliability in imbalanced classification tasks.

Contribution

The paper presents an intuitive framework for measuring metric sensitivity to class imbalance and uncovers a logarithmic behavior in this sensitivity, aiding better evaluation practices.

Findings

Metrics exhibit a logarithmic sensitivity to class imbalance.

Higher imbalance ratios are associated with lower metric sensitivity.

The framework helps avoid misinterpretations of metric comparability across imbalances.

Abstract

The class-imbalance issue is intrinsic to many real-world machine learning tasks, particularly to the rare-event classification problems. Although the impact and treatment of imbalanced data is widely known, the magnitude of a metric's sensitivity to class imbalance has attracted little attention. As a result, often the sensitive metrics are dismissed while their sensitivity may only be marginal. In this paper, we introduce an intuitive evaluation framework that quantifies metrics' sensitivity to the class imbalance. Moreover, we reveal an interesting fact that there is a logarithmic behavior in metrics' sensitivity meaning that the higher imbalance ratios are associated with the lower sensitivity of metrics. Our framework builds an intuitive understanding of the class-imbalance impact on metrics. We believe this can help avoid many common mistakes, specially the less-emphasized and incorrect assumption that all metrics' quantities are comparable under different class-imbalance ratios.