FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation

TL;DR

FairLoRA introduces a fairness-specific regularizer for Low Rank Adaptation in vision models, effectively reducing bias across subgroups without necessarily increasing model rank, and highlights the importance of multiple fairness metrics for evaluation.

Contribution

This paper presents the first fairness-based fine-tuning method using LoRA, introducing a regularizer that minimizes variance to improve fairness across data subgroups.

Findings

FairLoRA reduces bias without always increasing rank.

Performance disparities depend on model, dataset, and task.

Multiple fairness metrics are essential for comprehensive assessment.

Abstract

Recent advances in parameter-efficient fine-tuning methods, such as Low Rank Adaptation (LoRA), have gained significant attention for their ability to efficiently adapt large foundational models to various downstream tasks. These methods are appreciated for achieving performance comparable to full fine-tuning on aggregate-level metrics, while significantly reducing computational costs. To systematically address fairness in LLMs previous studies fine-tune on fairness specific data using a larger LoRA rank than typically used. In this paper, we introduce FairLoRA, a novel fairness-specific regularizer for LoRA aimed at reducing performance disparities across data subgroups by minimizing per-class variance in loss. To the best of our knowledge, we are the first to introduce a fairness based finetuning through LoRA. Our results demonstrate that the need for higher ranks to mitigate bias is not universal; it depends on factors such as the pre-trained model, dataset, and task. More importantly, we systematically evaluate FairLoRA across various vision models, including ViT, DiNO, and CLIP, in scenarios involving distribution shifts. We further emphasize the necessity of using multiple fairness metrics to obtain a holistic assessment of fairness, rather than relying solely on the metric optimized during training.