DemoBias: An Empirical Study to Trace Demographic Biases in Vision Foundation Models

TL;DR

This paper empirically investigates demographic biases in large vision language models used for biometric face recognition, revealing disparities across ethnicity, gender, and age groups, and evaluating the fairness of three popular models.

Contribution

It introduces DemoBias, a comprehensive empirical evaluation of demographic biases in LVLMs for biometric face recognition, using a balanced dataset and multiple fairness metrics.

Findings

PaliGemma and LLaVA show higher demographic disparities.

BLIP-2 demonstrates more consistent performance across groups.

The study highlights significant biases in popular LVLMs.

Abstract

Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities across various downstream tasks, including biometric face recognition (FR) with description. However, demographic biases remain a critical concern in FR, as these foundation models often fail to perform equitably across diverse demographic groups, considering ethnicity/race, gender, and age. Therefore, through our work DemoBias, we conduct an empirical evaluation to investigate the extent of demographic biases in LVLMs for biometric FR with textual token generation tasks. We fine-tuned and evaluated three widely used pre-trained LVLMs: LLaVA, BLIP-2, and PaliGemma on our own generated demographic-balanced dataset. We utilize several evaluation metrics, like group-specific BERTScores and the Fairness Discrepancy Rate, to quantify and trace the performance disparities. The experimental results deliver compelling insights into the fairness and reliability of LVLMs across diverse demographic groups. Our empirical study uncovered demographic biases in LVLMs, with PaliGemma and LLaVA exhibiting higher disparities for Hispanic/Latino, Caucasian, and South Asian groups, whereas BLIP-2 demonstrated comparably consistent. Repository: https://github.com/Sufianlab/DemoBias.