MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching

TL;DR

MVL-SIB is a comprehensive multilingual vision-language benchmark covering 205 languages, revealing that current LVLMs struggle with low-resource languages and multi-image tasks, highlighting areas for future improvement.

Contribution

Introduces MVL-SIB, the largest multilingual VL benchmark to date, and evaluates LVLMs, exposing their limitations in low-resource language understanding and multi-image processing.

Findings

LVLMs perform at chance level on low-resource languages like N'Koo.

VL support in LVLMs is disproportionately lower than text support for low-resource languages.

Open-weight LVLMs do not benefit from multi-image topic representations.

Abstract

Existing multilingual vision-language (VL) benchmarks often only cover a handful of languages. Consequently, evaluations of large vision-language models (LVLMs) predominantly target high-resource languages, underscoring the need for evaluation data for low-resource languages. To address this limitation, we introduce MVL-SIB, a massively multilingual vision-language benchmark that evaluates both cross-modal and text-only topical matching across 205 languages -- over 100 more than the most multilingual existing VL benchmarks encompass. We then benchmark a range of of open-weight LVLMs together with GPT-4o(-mini) on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic matching in lower-resource languages, performing no better than chance on languages like N'Koo. Our analysis further reveals that VL support in LVLMs declines disproportionately relative to textual support for lower-resource languages, as evidenced by comparison of cross-modal and text-only topical matching performance. We further observe that open-weight LVLMs do not benefit from representing a topic with more than one image, suggesting that these models are not yet fully effective at handling multi-image tasks. By correlating performance on MVL-SIB with other multilingual VL benchmarks, we highlight that MVL-SIB serves as a comprehensive probe of multilingual VL understanding in LVLMs.