Fairness Analysis of CLIP-Based Foundation Models for X-Ray Image Classification

TL;DR

This paper evaluates the fairness of CLIP-like models in X-ray image classification, revealing that despite accuracy improvements through fine-tuning, demographic fairness issues remain unaddressed, emphasizing the need for fairness interventions.

Contribution

It provides a comprehensive fairness analysis of CLIP-based models in medical imaging, highlighting persistent fairness issues despite performance enhancements from various fine-tuning methods.

Findings

Fine-tuning improves accuracy but not fairness.

Fairness disparities exist across demographics and disease categories.

Further fairness interventions are necessary for medical foundation models.

Abstract

X-ray imaging is pivotal in medical diagnostics, offering non-invasive insights into a range of health conditions. Recently, vision-language models, such as the Contrastive Language-Image Pretraining (CLIP) model, have demonstrated potential in improving diagnostic accuracy by leveraging large-scale image-text datasets. However, since CLIP was not initially designed for medical images, several CLIP-like models trained specifically on medical images have been developed. Despite their enhanced performance, issues of fairness - particularly regarding demographic attributes - remain largely unaddressed. In this study, we perform a comprehensive fairness analysis of CLIP-like models applied to X-ray image classification. We assess their performance and fairness across diverse patient demographics and disease categories using zero-shot inference and various fine-tuning techniques, including Linear Probing, Multilayer Perceptron (MLP), Low-Rank Adaptation (LoRA), and full fine-tuning. Our results indicate that while fine-tuning improves model accuracy, fairness concerns persist, highlighting the need for further fairness interventions in these foundational models.