World in a Frame: Understanding Culture Mixing as a New Challenge for Vision-Language Models

TL;DR

This paper investigates how large vision-language models handle culture mixing in images, revealing significant challenges and proposing robustness strategies to improve their cultural understanding and consistency.

Contribution

It introduces the CultureMix benchmark for evaluating LVLMs on culture mixing scenarios and demonstrates the effectiveness of supervised fine-tuning in enhancing model robustness.

Findings

LVLMs often fail to preserve cultural identities in mixed scenes.

Models heavily rely on backgrounds, reducing accuracy by 14%.

Supervised fine-tuning improves model consistency and reduces background sensitivity.

Abstract

In a globalized world, cultural elements from diverse origins frequently appear together within a single visual scene. We refer to these as culture mixing scenarios, yet how Large Vision-Language Models (LVLMs) perceive them remains underexplored. We investigate culture mixing as a critical challenge for LVLMs and examine how current models behave when cultural items from multiple regions appear together. To systematically analyze these behaviors, we construct CultureMix, a food Visual Question Answering (VQA) benchmark with 23k diffusion-generated, human-verified culture mixing images across four subtasks: (1) food-only, (2) food+food, (3) food+background, and (4) food+food+background. Evaluating 10 LVLMs, we find consistent failures to preserve individual cultural identities in mixed settings. Models show strong background reliance, with accuracy dropping 14% when cultural backgrounds are added to food-only baselines, and they produce inconsistent predictions for identical foods across different contexts. To address these limitations, we explore three robustness strategies. We find supervised fine-tuning using a diverse culture mixing dataset substantially improve model consistency and reduce background sensitivity. We call for increased attention to culture mixing scenarios as a critical step toward developing LVLMs capable of operating reliably in culturally diverse real-world environments.