PALO: A Polyglot Large Multimodal Model for 5B People

TL;DR

PALO is a large, multilingual vision-language model designed to provide visual reasoning in ten major languages, leveraging semi-automated translation and instruction tuning to enhance performance across diverse linguistic groups.

Contribution

This work introduces PALO, the first multilingual multimodal model trained on multiple languages with a scalable translation approach and a new benchmark for multilingual vision-language reasoning.

Findings

Substantial performance improvements over baselines.

Effective scalability across model sizes (1.7B, 7B, 13B).

Enhanced reasoning in underrepresented languages.

Abstract

In pursuit of more inclusive Vision-Language Models (VLMs), this study introduces a Large Multilingual Multimodal Model called PALO. PALO offers visual reasoning capabilities in 10 major languages, including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Urdu, and Japanese, that span a total of ~5B people (65% of the world population). Our approach involves a semi-automated translation approach to adapt the multimodal instruction dataset from English to the target languages using a fine-tuned Large Language Model, thereby ensuring high linguistic fidelity while allowing scalability due to minimal manual effort. The incorporation of diverse instruction sets helps us boost overall performance across multiple languages especially those that are underrepresented like Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three scales (1.7B, 7B and 13B parameters) to show the generalization and scalability where we observe substantial improvements compared to strong baselines. We also propose the first multilingual multimodal benchmark for the forthcoming approaches to evaluate their vision-language reasoning capabilities across languages. Code: https://github.com/mbzuai-oryx/PALO.