Data organization limits the predictability of binary classification

TL;DR

This paper establishes that the maximum achievable performance of binary classifiers is fundamentally limited by the data's inherent qualities, with theoretical bounds linked to dataset characteristics and class overlap.

Contribution

It introduces a theoretical framework for understanding data-imposed limits on binary classification performance and computes upper bounds for common evaluation metrics.

Findings

Theoretical upper bounds can be attained on real datasets.

Upper bounds for evaluation metrics are linked to dataset properties.

Class overlap influences the maximum achievable classifier performance.

Abstract

The structure of data organization is widely recognized as having a substantial influence on the efficacy of machine learning algorithms, particularly in binary classification tasks. Our research provides a theoretical framework suggesting that the maximum potential of binary classifiers on a given dataset is primarily constrained by the inherent qualities of the data. Through both theoretical reasoning and empirical examination, we employed standard objective functions, evaluative metrics, and binary classifiers to arrive at two principal conclusions. Firstly, we show that the theoretical upper bound of binary classification performance on actual datasets can be theoretically attained. This upper boundary represents a calculable equilibrium between the learning loss and the metric of evaluation. Secondly, we have computed the precise upper bounds for three commonly used evaluation metrics, uncovering a fundamental uniformity with our overarching thesis: the upper bound is intricately linked to the dataset's characteristics, independent of the classifier in use. Additionally, our subsequent analysis uncovers a detailed relationship between the upper limit of performance and the level of class overlap within the binary classification data. This relationship is instrumental for pinpointing the most effective feature subsets for use in feature engineering.