A Skew-Sensitive Evaluation Framework for Imbalanced Data Classification

TL;DR

This paper introduces a flexible evaluation framework for imbalanced data classification that accounts for class importance and skew, improving classifier assessment and training.

Contribution

It proposes a novel, general-purpose evaluation framework that is sensitive to class importance and skew, enhancing classifier evaluation and training.

Findings

Effective in evaluating and ranking classifiers

Improves classifier training on imbalanced data

Validated on real-world datasets from multiple domains

Abstract

Class distribution skews in imbalanced datasets may lead to models with prediction bias towards majority classes, making fair assessment of classifiers a challenging task. Metrics such as Balanced Accuracy are commonly used to evaluate a classifier's prediction performance under such scenarios. However, these metrics fall short when classes vary in importance. In this paper, we propose a simple and general-purpose evaluation framework for imbalanced data classification that is sensitive to arbitrary skews in class cardinalities and importances. Experiments with several state-of-the-art classifiers tested on real-world datasets from three different domains show the effectiveness of our framework - not only in evaluating and ranking classifiers, but also training them.