Rice-VL: Evaluating Vision-Language Models for Cultural Understanding Across ASEAN Countries

TL;DR

This paper introduces RICE-VL, a comprehensive benchmark for evaluating vision-language models' cultural understanding across ASEAN countries, revealing significant gaps and biases in current models.

Contribution

The paper presents RICE-VL, a new benchmark with diverse culturally annotated VQA and grounding tasks, and proposes SEA-LAVE to assess cultural alignment in VLMs.

Findings

VLMs show performance gaps in low-resource countries.

Models struggle with culturally significant visual grounding.

Current VLMs exhibit Western-centric biases.

Abstract

Vision-Language Models (VLMs) excel in multimodal tasks but often exhibit Western-centric biases, limiting their effectiveness in culturally diverse regions like Southeast Asia (SEA). To address this, we introduce RICE-VL, a novel benchmark evaluating VLM cultural understanding across 11 ASEAN countries. RICE-VL includes over 28,000 human-curated Visual Question Answering (VQA) samples -- covering True or False, Fill-in-the-Blank, and open-ended formats -- and 1,000 image-bounding box pairs for Visual Grounding, annotated by culturally informed experts across 14 sub-ground categories. We propose SEA-LAVE, an extension of the LAVE metric, assessing textual accuracy, cultural alignment, and country identification. Evaluations of six open- and closed-source VLMs reveal significant performance gaps in low-resource countries and abstract cultural domains. The Visual Grounding task tests models' ability to localize culturally significant elements in complex scenes, probing spatial and contextual accuracy. RICE-VL exposes limitations in VLMs' cultural comprehension and highlights the need for inclusive model development to better serve diverse global populations.