Addressing Bias in VLMs for Glaucoma Detection Without Protected Attribute Supervision

TL;DR

This paper proposes an unsupervised, label-free debiasing method for vision-language models in glaucoma detection, reducing demographic disparities without protected attribute labels by clustering and reweighting hard examples.

Contribution

It introduces an attribute-agnostic debiasing framework that infers subgroups via clustering and adaptively reweights training to improve fairness in glaucoma screening.

Findings

Reduces subgroup disparities in glaucoma detection.

Improves fairness metrics like EOD and subgroup AUC.

Demonstrates effectiveness on the Harvard FairVLMed dataset.

Abstract

Vision-Language Models (VLMs) have achieved remarkable success on multimodal tasks such as image-text retrieval and zero-shot classification, yet they can exhibit demographic biases even when explicit protected attributes are absent during training. In this work, we focus on automated glaucoma screening from retinal fundus images, a critical application given that glaucoma is a leading cause of irreversible blindness and disproportionately affects underserved populations. Building on a reweighting-based contrastive learning framework, we introduce an attribute-agnostic debiasing method that (i) infers proxy subgroups via unsupervised clustering of image-image embeddings, (ii) computes gradient-similarity weights between the CLIP-style multimodal loss and a SimCLR-style image-pair contrastive loss, and (iii) applies these weights in a joint, top-$k$ weighted objective to upweight underperforming clusters. This label-free approach adaptively targets the hardest examples, thereby reducing subgroup disparities. We evaluate our method on the Harvard FairVLMed glaucoma subset, reporting Equalized Odds Distance (EOD), Equalized Subgroup AUC (ES AUC), and Groupwise AUC to demonstrate equitable performance across inferred demographic subgroups.