Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation

TL;DR

This paper evaluates the cultural competence of vision-language models in multimodal story generation, revealing both their potential for cultural adaptation and significant limitations across architectures and metrics.

Contribution

It introduces the first systematic framework for assessing cultural awareness in multimodal VLMs through story generation tasks.

Findings

Models show rich culturally-specific vocabulary usage.

Cultural competence varies significantly across architectures.

Automated metrics often contradict human assessments.

Abstract

As Vision-Language Models (VLMs) achieve widespread deployment across diverse cultural contexts, ensuring their cultural competence becomes critical for responsible AI systems. While prior work has evaluated cultural awareness in text-only models and VLM object recognition tasks, no research has systematically assessed how VLMs adapt outputs when cultural identity cues are embedded in both textual prompts and visual inputs during generative tasks. We present the first comprehensive evaluation of VLM cultural competence through multimodal story generation, developing a novel multimodal framework that perturbs cultural identity and evaluates 5 contemporary VLMs on a downstream task: story generation. Our analysis reveals significant cultural adaptation capabilities, with rich culturally-specific vocabulary spanning names, familial terms, and geographic markers. However, we uncover concerning limitations: cultural competence varies dramatically across architectures, some models exhibit inverse cultural alignment, and automated metrics show architectural bias contradicting human assessments. Cross-modal evaluation shows that culturally distinct outputs are indeed detectable through visual-semantic similarity (28.7% within-nationality vs. 0.2% cross-nationality recall), yet visual-cultural understanding remains limited. In essence, we establish the promise and challenges of cultural competence in multimodal AI. We publicly release our codebase and data: https://github.com/ArkaMukherjee0/mmCultural