Multilingual Training and Evaluation Resources for Vision-Language Models

TL;DR

This paper introduces multilingual training and evaluation resources for Vision-Language Models across five European languages, enhancing training datasets and benchmarks to improve model performance in multilingual settings.

Contribution

The work provides a comprehensive multilingual dataset and benchmarks for VLMs, created through regeneration-translation, and demonstrates the benefits of multilingual training.

Findings

Multilingual data improves VLM performance on non-English benchmarks.

Using multilingual, multimodal examples benefits VLM training across languages.

Positive transfer observed from multilingual training to English benchmarks.

Abstract

Vision Language Models (VLMs) achieved rapid progress in the recent years. However, despite their growth, VLMs development is heavily grounded on English, leading to two main limitations: (i) the lack of multilingual and multimodal datasets for training, and (ii) the scarcity of comprehensive evaluation benchmarks across languages. In this work, we address these gaps by introducing a new comprehensive suite of resources for VLMs training and evaluation spanning five European languages (English, French, German, Italian, and Spanish). We adopt a regeneration-translation paradigm that produces high-quality cross-lingual resources by combining curated synthetic generation and manual annotation. Specifically, we build Multi-PixMo, a training corpus obtained regenerating examples from Pixmo pre-existing datasets with permissively licensed models: PixMo-Cap, PixMo-AskModelAnything, and CoSyn-400k. On the evaluation side, we construct a set of multilingual benchmarks derived translating widely used English datasets (MMbench, ScienceQA, MME, POPE, AI2D). We assess the quality of these resources through qualitative and quantitative human analyses, measuring inter-annotator agreement. Additionally, we perform ablation studies to demonstrate the impact of multilingual data, with respect to English only, in VLMs training. Experiments, comprising 3 different models show that using multilingual, multimodal examples for training VLMs aids is consistently beneficial on non-English benchmarks, with positive transfer to English as well.