VULCA-Bench: A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding

TL;DR

VULCA-Bench is a comprehensive multicultural vision-language benchmark designed to evaluate models' deep cultural understanding through 7,410 image-critique pairs across eight cultures, emphasizing higher-order reasoning beyond surface perception.

Contribution

It introduces a new benchmark with a five-layer framework and expert annotations to assess cultural understanding in vision-language models, addressing limitations of existing benchmarks.

Findings

Higher-layer reasoning (L3-L5) is more challenging than visual perception (L1-L2).

The dataset includes 7,410 image-critique pairs across eight cultures.

Evaluation scripts and tools are publicly available.

Abstract

We introduce VULCA-Bench, a multicultural art-critique benchmark for evaluating Vision-Language Models' (VLMs) cultural understanding beyond surface-level visual perception. Existing VLM benchmarks predominantly measure L1-L2 capabilities (object recognition, scene description, and factual question answering) while under-evaluate higher-order cultural interpretation. VULCA-Bench contains 7,410 matched image-critique pairs spanning eight cultural traditions, with Chinese-English bilingual coverage. We operationalise cultural understanding using a five-layer framework (L1-L5, from Visual Perception to Philosophical Aesthetics), instantiated as 225 culture-specific dimensions and supported by expert-written bilingual critiques. Our pilot results indicate that higher-layer reasoning (L3-L5) is consistently more challenging than visual and technical analysis (L1-L2). The dataset, evaluation scripts, and annotation tools are available under CC BY 4.0 at https://github.com/yha9806/VULCA-Bench.