Intersectional Fairness in Vision-Language Models for Medical Image Disease Classification

TL;DR

This paper introduces CMAC-MMD, a training framework that standardizes diagnostic confidence across intersectional patient groups in vision-language models, reducing bias without needing sensitive demographic data during inference.

Contribution

The study presents a novel training method, CMAC-MMD, that improves fairness and diagnostic accuracy across diverse subgroups in medical imaging models without compromising overall performance.

Findings

Reduced intersectional missed diagnosis gap in dermatology from 0.50 to 0.26

Improved AUC from 0.94 to 0.97 in skin lesion classification

Lowered TPR gap in glaucoma detection from 0.41 to 0.31

Abstract

Medical artificial intelligence (AI) systems, particularly multimodal vision-language models (VLM), often exhibit intersectional biases where models are systematically less confident in diagnosing marginalised patient subgroups. Such bias can lead to higher rates of inaccurate and missed diagnoses due to demographically skewed data and divergent distributions of diagnostic certainty. Current fairness interventions frequently fail to address these gaps or compromise overall diagnostic performance to achieve statistical parity among the subgroups. In this study, we developed Cross-Modal Alignment Consistency (CMAC-MMD), a training framework that standardises diagnostic certainty across intersectional patient subgroups. Unlike traditional debiasing methods, this approach equalises the model's decision confidence without requiring sensitive demographic data during clinical inference. We evaluated this approach using 10,015 skin lesion images (HAM10000) with external validation on 12,000 images (BCN20000), and 10,000 fundus images for glaucoma detection (Harvard-FairVLMed), stratifying performance by intersectional age, gender, and race attributes. In the dermatology cohort, the proposed method reduced the overall intersectional missed diagnosis gap (difference in True Positive Rate, $\Delta$TPR) from 0.50 to 0.26 while improving the overall Area Under the Curve (AUC) from 0.94 to 0.97 compared to standard training. Similarly, for glaucoma screening, the method reduced $\Delta$TPR from 0.41 to 0.31, achieving a better AUC of 0.72 (vs. 0.71 baseline). This establishes a scalable framework for developing high-stakes clinical decision support systems that are both accurate and can perform equitably across diverse patient subgroups, ensuring reliable performance without increasing privacy risks.