DuetFair: Coupling Inter- and Intra-Subgroup Robustness for Fair Medical Image Segmentation

TL;DR

DuetFair introduces a dual-axis fairness framework for medical image segmentation that addresses both inter- and intra-subgroup disparities, improving equity and robustness across diverse patient groups.

Contribution

The paper proposes FairDRO, a novel method combining distribution-aware mixture-of-experts and subgroup-conditioned DRO to enhance fairness and intra-group robustness.

Findings

FairDRO achieves the best equity-scaled performance on Harvard-FairSeg.

It improves worst-case subgroup performance on HAM10000 across age and race groups.

On a 3D radiotherapy cohort, FairDRO increases worst-group Dice by up to 7.4%.

Abstract

Medical image segmentation models can perform unevenly across subgroups. Most existing fairness methods focus on improving average subgroup performance, implicitly treating each subgroup as internally homogeneous. However, this can hide difficult cases within a subgroup, where high-loss samples are obscured by the subgroup mean. We call this problem \textbf{intra-group hidden failure}. To solve this, we propose \textbf{DuetFair} mechanism, a dual-axis fairness framework that jointly considers inter-subgroup adaptation and intra-subgroup robustness. Based on DuetFair, we introduce \textbf{FairDRO}, which combines distribution-aware mixture-of-experts (dMoE) with subgroup-conditioned distributionally robust optimization (DRO) loss aggregation. This design allows the model to adapt across subgroups while also reducing hidden failures within each subgroup. We evaluate FairDRO on three medical image segmentation benchmarks with varying degrees of within-group heterogeneity. FairDRO achieves the best equity-scaled performance on Harvard-FairSeg and improves worst-case subgroup performance on HAM10000 under both age- and race-based grouping schemes. On the 3D radiotherapy target cohort, FairDRO further improves worst-group Dice by 3.5 points ($\uparrow 6.0\%$) under the tumor-stage grouping and by 4.1 points ($\uparrow 7.4\%$) under the institution grouping over the strongest baseline.