Worst-Group Equalized Odds Regularization for Multi-Attribute Fair Medical Image Classification

TL;DR

This paper introduces a regularization method to improve fairness in medical image classification by reducing disparities across demographic groups at the inference stage.

Contribution

It proposes a worst-group equalized-odds margin regularizer that targets subgroup-level fairness without sacrificing overall diagnostic accuracy.

Findings

Reduces disparities in Equalized Odds and Opportunity across demographic groups.

Maintains high diagnostic AUC while improving fairness.

Effective across multiple medical imaging datasets with multi-label settings.

Abstract

Diagnostic performance in medical AI varies systematically across demographic groups, yet subgroup AUC can mask clinically important disparities. At a fixed inference-time operating point, some groups may exhibit over-diagnostic behaviour, characterized by elevated true and false positive rates, while others show under-diagnostic patterns with reduced true and false positive rates. These opposing tendencies can cancel in aggregate AUCs while producing meaningful inequities in clinical decision-making. Motivated by the need to assess and mitigate such disparities at the operating point and across multiple demographic attributes simultaneously, we propose a worst-group equalized-odds margin regularizer. The proposed regularizer explicitly targets subgroup-level deviations on both the true positive and false positive sides at inference. At each update, the method identifies subgroups defined by explicit demographic attributes (e.g., age, sex, and race) that exhibit the most extreme margin deviations and applies a unified penalty, enabling fairness optimization across multiple demographic axes without requiring explicit intersectional constraints. Across two medical imaging datasets in realistic multi-label settings, our method consistently reduces disparities in Equalized Odds and Equalized Opportunity with minimal impact on AUC, preserving diagnostic performance while improving fairness.