Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration

TL;DR

This paper introduces K-Viscuit, a culturally inclusive benchmark for vision-language models focusing on Korean culture, created through a semi-automated human-AI collaboration process to evaluate and improve model understanding of cultural nuances.

Contribution

We propose a semi-automated framework for building cultural VLM benchmarks and demonstrate its effectiveness with the K-Viscuit dataset, highlighting gaps in current models' cultural understanding.

Findings

Open-source models underperform proprietary models on Korean cultural questions.

The framework effectively reduces manual effort in dataset creation.

External knowledge augmentation improves VLM performance.

Abstract

To create culturally inclusive vision-language models (VLMs), developing a benchmark that tests their ability to address culturally relevant questions is essential. Existing approaches typically rely on human annotators, making the process labor-intensive and creating a cognitive burden in generating diverse questions. To address this, we propose a semi-automated framework for constructing cultural VLM benchmarks, specifically targeting multiple-choice QA. This framework combines human-VLM collaboration, where VLMs generate questions based on guidelines, a small set of annotated examples, and relevant knowledge, followed by a verification process by native speakers. We demonstrate the effectiveness of this framework through the creation of \texttt{K-Viscuit}, a dataset focused on Korean culture. Our experiments on this dataset reveal that open-source models lag behind proprietary ones in understanding Korean culture, highlighting key areas for improvement. We also present a series of further analyses, including human evaluation, augmenting VLMs with external knowledge, and the evaluation beyond multiple-choice QA. Our dataset is available at https://huggingface.co/datasets/ddehun/k-viscuit.