Understanding Museum Exhibits using Vision-Language Reasoning

TL;DR

This paper introduces a large-scale dataset of museum exhibit images and questions, trains vision-language models on it, and benchmarks their ability to answer complex, context-rich museum-related queries, highlighting the importance of domain-specific fine-tuning.

Contribution

The creation of a comprehensive museum exhibit dataset and the evaluation of vision-language models' performance on real-world museum inquiry tasks, emphasizing the need for domain-specific fine-tuning.

Findings

Large vision-language models outperform smaller models in historical reasoning.

Fine-tuned models significantly surpass SOTA in attribute-based questions.

Expert-labeled dataset ensures high-quality, practical annotations.

Abstract

Museums serve as repositories of cultural heritage and historical artifacts from diverse epochs, civilizations, and regions, preserving well-documented collections that encapsulate vast knowledge, which, when systematically structured into large-scale datasets, can train specialized models. Visitors engage with exhibits through curiosity and questions, making expert domain-specific models essential for interactive query resolution and gaining historical insights. Understanding exhibits from images requires analyzing visual features and linking them to historical knowledge to derive meaningful correlations. We facilitate such reasoning by (a) collecting and curating a large-scale dataset of 65M images and 200M question-answer pairs for exhibits from all around the world; (b) training large vision-language models (VLMs) on the collected dataset; (c) benchmarking their ability on five visual question answering tasks, specifically designed to reflect real-world inquiries and challenges observed in museum settings. The complete dataset is labeled by museum experts, ensuring the quality and the practical significance of the labels. We train two VLMs from different categories: BLIP with vision-language aligned embeddings, but lacking the expressive power of large language models, and the LLaVA model, a powerful instruction-tuned LLM enriched with vision-language reasoning capabilities. Through extensive experiments, we find that while both model types effectively answer visually grounded questions, large vision-language models excel in queries requiring deeper historical context and reasoning. We further demonstrate the necessity of fine-tuning models on large-scale domain-specific datasets by showing that our fine-tuned models significantly outperform current SOTA VLMs in answering questions related to specific attributes, highlighting their limitations in handling complex, nuanced queries.