DUCX: Decomposing Unfairness in Tool-Using Chest X-ray Agents

TL;DR

This paper systematically audits fairness in tool-using chest X-ray agents, revealing persistent demographic disparities at multiple stages and emphasizing the need for process-level fairness evaluation and mitigation.

Contribution

It introduces a stage-wise fairness decomposition method to localize bias sources in chest X-ray agents, highlighting disparities beyond end-to-end performance.

Findings

Demographic gaps persist in end-to-end performance, with up to 20.79% equalized odds.

Intermediate behaviors show subgroup disparities not predictable from end-to-end metrics.

Conditioned on segmentation-tool availability, subgroup utility gaps reach as high as 50%.

Abstract

Tool-using medical agents can improve chest X-ray question answering by orchestrating specialized vision and language modules, but this added pipeline complexity also creates new pathways for demographic bias beyond standalone models. We present ours (Decomposing Unfairness in Chest X-ray agents), a systematic audit of chest X-ray agents instantiated with MedRAX. To localize where disparities arise, we introduce a stage-wise fairness decomposition that separates end-to-end bias from three agent-specific sources: tool exposure bias (utility gaps conditioned on tool presence), tool transition bias (subgroup differences in tool-routing patterns), and model reasoning bias (subgroup differences in synthesis behaviors). Extensive experiments on tool-used based agentic frameworks across five driver backbones reveal that (i) demographic gaps persist in end-to-end performance, with equalized odds up to 20.79%, and the lowest fairness-utility tradeoff down to 28.65%, and (ii) intermediate behaviors, tool usage, transition patterns, and reasoning traces exhibit distinct subgroup disparities that are not predictable from end-to-end evaluation alone (e.g., conditioned on segmentation-tool availability, the subgroup utility gap reaches as high as 50%). Our findings underscore the need for process-level fairness auditing and debiasing to ensure the equitable deployment of clinical agentic systems. Code is available here: https://anonymous.4open.science/r/DUCK-E5FE/README.md