Uncovering Cultural Representation Disparities in Vision-Language Models

TL;DR

This paper evaluates cultural biases in vision-language models by testing their accuracy on a country identification task across diverse datasets and prompting strategies, revealing significant disparities influenced by training data biases.

Contribution

It introduces a comprehensive evaluation of cultural biases in VLMs using the Country211 dataset and various prompting methods, highlighting how data distribution affects model fairness.

Findings

VLMs show significant accuracy disparities across countries.

Prompting strategies influence model bias and performance.

Training data biases impact model generalization across cultures.

Abstract

Vision-Language Models (VLMs) have demonstrated impressive capabilities across a range of tasks, yet concerns about their potential biases exist. This work investigates the extent to which prominent VLMs exhibit cultural biases by evaluating their performance on an image-based country identification task at a country level. Utilizing the geographically diverse Country211 dataset, we probe several large vision language models (VLMs) under various prompting strategies: open-ended questions, multiple-choice questions (MCQs) including challenging setups like multilingual and adversarial settings. Our analysis aims to uncover disparities in model accuracy across different countries and question formats, providing insights into how training data distribution and evaluation methodologies might influence cultural biases in VLMs. The findings highlight significant variations in performance, suggesting that while VLMs possess considerable visual understanding, they inherit biases from their pre-training data and scale that impact their ability to generalize uniformly across diverse global contexts.