Freeze and Reveal: Exposing Modality Bias in Vision-Language Models

TL;DR

This paper investigates the sources of gender bias in vision-language models, proposing novel metrics and methods to reduce bias efficiently by targeting vision or text components, thereby improving fairness and accuracy.

Contribution

It introduces a new metric, Degree of Stereotypicality, and a debiasing method, DAUDoS, to effectively mitigate bias with minimal data and computational resources.

Findings

CDA reduces gender bias gap by 6%

DAUDoS reduces bias gap by 3% using less data

Vision encoder (CLIP) is more biased than text encoder (PaliGemma2)

Abstract

Vision Language Models achieve impressive multi-modal performance but often inherit gender biases from their training data. This bias might be coming from both the vision and text modalities. In this work, we dissect the contributions of vision and text backbones to these biases by applying targeted debiasing using Counterfactual Data Augmentation and Task Vector methods. Inspired by data-efficient approaches in hate-speech classification, we introduce a novel metric, Degree of Stereotypicality and a corresponding debiasing method, Data Augmentation Using Degree of Stereotypicality - DAUDoS, to reduce bias with minimal computational cost. We curate a gender annotated dataset and evaluate all methods on VisoGender benchmark to quantify improvements and identify dominant source of bias. Our results show that CDA reduces the gender gap by 6% and DAUDoS by 3% but using only one-third of the data. Both methods also improve the model's ability to correctly identify gender in images by 3%, with DAUDoS achieving this improvement using only almost one-third of training data. From our experiment's, we observed that CLIP's vision encoder is more biased whereas PaliGemma2's text encoder is more biased. By identifying whether bias stems more from vision or text encoders, our work enables more targeted and effective bias mitigation strategies in future multi-modal systems.