Fairness-Aware Fine-Tuning of Vision-Language Models for Medical Glaucoma Diagnosis

TL;DR

This paper presents a fairness-aware fine-tuning method for vision-language models in medical glaucoma diagnosis, significantly reducing demographic disparities while maintaining high accuracy with minimal additional parameters.

Contribution

It introduces a differentiable MaxAccGap loss and three fine-tuning methods that improve fairness in medical VLMs with parameter-efficient adaptation.

Findings

GR-LoRA reduces accuracy disparities by 69%

Maintains 53.15% overall accuracy

Requires only 0.24% trainable parameters

Abstract

Vision-language models achieve expert-level performance on medical imaging tasks but exhibit significant diagnostic accuracy disparities across demographic groups. We introduce fairness-aware Low-Rank Adaptation for medical VLMs, combining parameter efficiency with explicit fairness optimization. Our key algorithmic contribution is a differentiable MaxAccGap loss that enables end-to-end optimization of accuracy parity across demographic groups. We propose three methods: FR-LoRA integrates MaxAccGap regularization into the training objective, GR-LoRA applies inverse frequency weighting to balance gradient contributions, and Hybrid-LoRA combines both mechanisms. Evaluated on 10,000 glaucoma fundus images, GR-LoRA reduces diagnostic accuracy disparities by 69% while maintaining 53.15% overall accuracy. Ablation studies reveal that strong regularization strength achieves optimal fairness with minimal accuracy trade-off, and race-specific optimization yields 60% disparity reduction. Our approach requires only 0.24% trainable parameters, enabling practical deployment of fair medical AI in resource-constrained healthcare settings.