Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages

TL;DR

Pangea is a fully open, multilingual multimodal large language model trained on 39 languages, demonstrating superior performance across diverse cultural and linguistic contexts, and promoting inclusivity in AI.

Contribution

Introduces Pangea, a novel multilingual multimodal LLM trained on a diverse dataset, with a comprehensive evaluation suite, advancing inclusive AI development.

Findings

Pangea outperforms existing open-source models in multilingual tasks.

English data proportion and cultural relevance impact model performance.

Open-sourcing promotes inclusive AI development.

Abstract

Despite recent advances in multimodal large language models (MLLMs), their development has predominantly focused on English- and western-centric datasets and tasks, leaving most of the world's languages and diverse cultural contexts underrepresented. This paper introduces Pangea, a multilingual multimodal LLM trained on PangeaIns, a diverse 6M instruction dataset spanning 39 languages. PangeaIns features: 1) high-quality English instructions, 2) carefully machine-translated instructions, and 3) culturally relevant multimodal tasks to ensure cross-cultural coverage. To rigorously assess models' capabilities, we introduce PangeaBench, a holistic evaluation suite encompassing 14 datasets covering 47 languages. Results show that Pangea significantly outperforms existing open-source models in multilingual settings and diverse cultural contexts. Ablation studies further reveal the importance of English data proportions, language popularity, and the number of multimodal training samples on overall performance. We fully open-source our data, code, and trained checkpoints, to facilitate the development of inclusive and robust multilingual MLLMs, promoting equity and accessibility across a broader linguistic and cultural spectrum.