Correcting Performance Estimation Bias in Imbalanced Classification with Minority Subconcepts

TL;DR

This paper proposes a new evaluation metric, predicted-weighted balanced accuracy (pBA), that accounts for within-class heterogeneity in imbalanced classification, improving stability and interpretability.

Contribution

It introduces a practical, uncertainty-aware evaluation method using predicted posterior probabilities to correct performance bias caused by subconcept heterogeneity.

Findings

pBA provides more stable performance estimates than unweighted scores.

Unweighted scores can be misleading under within-class heterogeneity.

The method improves evaluation stability across diverse datasets.

Abstract

Class-level evaluation can conceal substantial performance disparities across subconcepts within the same class, causing models that perform well on average to fail on specific subpopulations. Prior work has shown that common evaluation measures for imbalanced classification are biased toward larger minority subconcepts and that utility-based reweighting using true subconcept labels can mitigate this bias; however, such labels are rarely available at test time. We introduce a practical utility-weighted evaluation that replaces unavailable subconcept labels with predicted posterior probabilities from a multiclass subconcept model. Evaluation weights are defined as the expected utility under this posterior, yielding a soft, uncertainty-aware metric we call predicted-weighted balanced accuracy (pBA). Experiments on tabular benchmarks as well as medical-imaging and text datasets show that unweighted scores can be misleading under within-class heterogeneity, while pBA provides more stable and interpretable assessments when subconcept distributions are uneven but not pathological. Our code is available at: https://anonymous.4open.science/r/correcting-bias-imbalance-9C6C/.