Decomposing Global AUC into Cluster-Level Contributions for Localized Model Diagnostics

TL;DR

This paper introduces a method to decompose the global AUC into cluster-level contributions, enabling detailed diagnostics of classifier performance within and across data subgroups for better model risk management.

Contribution

It provides a formal decomposition of AUC into intra- and inter-cluster components, facilitating granular model diagnostics and subgroup analysis.

Findings

Decomposition of AUC into cluster-level contributions

Comparison of AUC with additive metrics like Brier score and log loss

Enhanced model validation through localized performance insights

Abstract

The Area Under the ROC Curve (AUC) is a widely used performance metric for binary classifiers. However, as a global ranking statistic, the AUC aggregates model behavior over the entire dataset, masking localized weaknesses in specific subpopulations. In high-stakes applications such as credit approval and fraud detection, these weaknesses can lead to financial risk or operational failures. In this paper, we introduce a formal decomposition of global AUC into intra- and inter-cluster components. This allows practitioners to evaluate classifier performance within and across clusters of data, enabling granular diagnostics and subgroup analysis. We also compare the AUC with additive performance metrics such as the Brier score and log loss, which support decomposability and direct attribution. Our framework enhances model development and validation practice by providing additional insights to detect model weakness for model risk management.