Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images

TL;DR

This paper introduces a new cross-cultural benchmark to evaluate vision-language models' ability to infer structured cultural metadata from images, revealing their current limitations and performance variability.

Contribution

It presents a novel benchmark and evaluation framework for assessing VLMs' cultural reasoning and structured metadata inference from images.

Findings

Models capture fragmented cultural signals.

Performance varies significantly across cultures.

Current VLMs show weak grounding in cultural metadata inference.

Abstract

Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage. However, inferring structured cultural metadata (e.g., creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural benchmark for this task and evaluate VLMs using an LLM-as-Judge framework that measures semantic alignment with reference annotations. To assess cultural reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.