Cultural Awareness in Vision-Language Models: A Cross-Country Exploration

TL;DR

This paper introduces a framework to evaluate cultural biases in vision-language models by analyzing their associations with race, gender, and physical traits across different countries, revealing persistent stereotypes.

Contribution

It presents three novel retrieval-based tasks to systematically assess how VLMs encode cultural differences and biases related to race, traits, and physical features across countries.

Findings

VLMs show persistent biases linking physical traits to specific countries.

Models reinforce societal stereotypes related to race and personal traits.

Cultural biases are encoded in visual representations within VLMs.

Abstract

Vision-Language Models (VLMs) are increasingly deployed in diverse cultural contexts, yet their internal biases remain poorly understood. In this work, we propose a novel framework to systematically evaluate how VLMs encode cultural differences and biases related to race, gender, and physical traits across countries. We introduce three retrieval-based tasks: (1) Race to Country retrieval, which examines the association between individuals from specific racial groups (East Asian, White, Middle Eastern, Latino, South Asian, and Black) and different countries; (2) Personal Traits to Country retrieval, where images are paired with trait-based prompts (e.g., Smart, Honest, Criminal, Violent) to investigate potential stereotypical associations; and (3) Physical Characteristics to Country retrieval, focusing on visual attributes like skinny, young, obese, and old to explore how physical appearances are culturally linked to nations. Our findings reveal persistent biases in VLMs, highlighting how visual representations may inadvertently reinforce societal stereotypes.