Fairness in Multi-modal Medical Diagnosis with Demonstration Selection

TL;DR

This paper introduces FADS, a demographically balanced demonstration selection method for multimodal large language models, significantly reducing biases in medical image reasoning without extensive fine-tuning.

Contribution

It proposes a novel fairness-aware demonstration selection technique using clustering, improving fairness in medical diagnosis models without additional training.

Findings

FADS reduces demographic disparities in medical imaging tasks.

Maintains high accuracy while improving fairness.

Scalable and data-efficient approach for equitable medical AI.

Abstract

Multimodal large language models (MLLMs) have shown strong potential for medical image reasoning, yet fairness across demographic groups remains a major concern. Existing debiasing methods often rely on large labeled datasets or fine-tuning, which are impractical for foundation-scale models. We explore In-Context Learning (ICL) as a lightweight, tuning-free alternative for improving fairness. Through systematic analysis, we find that conventional demonstration selection (DS) strategies fail to ensure fairness due to demographic imbalance in selected exemplars. To address this, we propose Fairness-Aware Demonstration Selection (FADS), which builds demographically balanced and semantically relevant demonstrations via clustering-based sampling. Experiments on multiple medical imaging benchmarks show that FADS consistently reduces gender-, race-, and ethnicity-related disparities while maintaining strong accuracy, offering an efficient and scalable path toward fair medical image reasoning. These results highlight the potential of fairness-aware in-context learning as a scalable and data-efficient solution for equitable medical image reasoning.