CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation

TL;DR

CVLUE introduces a culturally relevant Chinese vision-language benchmark dataset to evaluate and improve models' understanding of Chinese culture, addressing biases in existing datasets and revealing performance gaps.

Contribution

The paper presents CVLUE, a new Chinese VL dataset created with native speaker input, and provides baseline evaluations highlighting cultural knowledge gaps in current VLMs.

Findings

Existing VLMs lack Chinese cultural knowledge.

Fine-tuning on Chinese datasets improves VLMs' cultural understanding.

CVLUE enables more accurate evaluation of Chinese vision-language models.

Abstract

Despite the rapid development of Chinese vision-language models (VLMs), most existing Chinese vision-language (VL) datasets are constructed on Western-centric images from existing English VL datasets. The cultural bias in the images makes these datasets unsuitable for evaluating VLMs in Chinese culture. To remedy this issue, we present a new Chinese Vision- Language Understanding Evaluation (CVLUE) benchmark dataset, where the selection of object categories and images is entirely driven by Chinese native speakers, ensuring that the source images are representative of Chinese culture. The benchmark contains four distinct VL tasks ranging from image-text retrieval to visual question answering, visual grounding and visual dialogue. We present a detailed statistical analysis of CVLUE and provide a baseline performance analysis with several open-source multilingual VLMs on CVLUE and its English counterparts to reveal their performance gap between English and Chinese. Our in-depth category-level analysis reveals a lack of Chinese cultural knowledge in existing VLMs. We also find that fine-tuning on Chinese culture-related VL datasets effectively enhances VLMs' understanding of Chinese culture.