Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts

TL;DR

This paper evaluates demographic fairness in large vision-language models, revealing persistent biases and proposing a multi-modal Chain-of-thought strategy to mitigate unfairness across diverse demographic groups.

Contribution

It introduces a comprehensive fairness evaluation framework for LVLMs and proposes a novel multi-modal Chain-of-thought method for addressing biases.

Findings

LVLMs exhibit significant fairness disparities across demographic attributes.

Zero-shot prompting reveals persistent bias issues in LVLMs.

Proposed Chain-of-thought strategy improves fairness and transparency.

Abstract

Large vision-language models (LVLMs) have recently achieved significant progress, demonstrating strong capabilities in open-world visual understanding. However, it is not yet clear how LVLMs address demographic biases in real life, especially the disparities across attributes such as gender, skin tone, age and race. In this paper, We empirically investigate \emph{visual fairness} in several mainstream LVLMs by auditing their performance disparities across demographic attributes using public fairness benchmark datasets (e.g., FACET, UTKFace). Our fairness evaluation framework employs direct and single-choice question prompt on visual question-answering/classification tasks. Despite advancements in visual understanding, our zero-shot prompting results show that both open-source and closed-source LVLMs continue to exhibit fairness issues across different prompts and demographic groups. Furthermore, we propose a potential multi-modal Chain-of-thought (CoT) based strategy for unfairness mitigation, applicable to both open-source and closed-source LVLMs. This approach enhances transparency and offers a scalable solution for addressing fairness, providing a solid foundation for future research and practical efforts in unfairness mitigation. The dataset and code used in this study are publicly available at this GitHub Repository.