TowerVision: Understanding and Improving Multilinguality in Vision-Language Models

TL;DR

This paper introduces TowerVision, a family of multilingual vision-language models built on Tower+, demonstrating improved cross-lingual performance and cultural understanding in multimodal tasks, supported by extensive empirical analysis and new datasets.

Contribution

The paper presents TowerVision, a novel multilingual VLM family, along with insights on multilingual training data, model initialization, and a new curated dataset, advancing multilingual multimodal understanding.

Findings

Multilingual training data enhances cross-lingual generalization.

Instruction-tuned LLMs are not always optimal for initialization.

Visual and cultural context improves performance on culturally grounded tasks.

Abstract

Despite significant advances in vision-language models (VLMs), most existing work follows an English-centric design process, limiting their effectiveness in multilingual settings. In this work, we provide a comprehensive empirical study analyzing the impact of several multilingual design choices, such as training data composition, encoder selection, and text backbones. The result is TowerVision, a family of open multilingual VLMs for both image-text and video-text tasks, built upon the multilingual text-only model Tower+. TowerVision achieves competitive performance on multiple multimodal multilingual benchmarks and shows particular strength in culturally grounded tasks and multimodal translation. By incorporating visual and cultural context during fine-tuning, our models surpass existing approaches trained on substantially larger datasets, as demonstrated on ALM-Bench and Multi30K (image tasks) and ViMUL-Bench (video tasks). Alongside the models, we release VisionBlocks, a high-quality, curated vision-language dataset. Our findings highlight that multilingual vision-language training data substantially improves cross-lingual generalization -- both from high-resource to underrepresented languages and vice versa -- and that instruction-tuned LLMs are not always the optimal initialization point. To support further research, we publicly release all models, data, and training recipes.