Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation

TL;DR

This paper systematically evaluates vision-language translation models, introduces a high-quality multilingual dataset, proposes a new evaluation metric, and develops a balanced fine-tuning strategy to improve model adaptation.

Contribution

It presents a comprehensive analysis of VLT, introduces AibTrans dataset, proposes Density-Aware Evaluation, and offers a novel fine-tuning approach for better multilingual VLT performance.

Findings

Existing datasets lack semantic and cultural fidelity.

Fine-tuning on high-resource languages can harm cross-lingual performance.

Density-Aware Evaluation provides more reliable translation quality metrics.

Abstract

Vision-Language Translation (VLT) is a challenging task that requires accurately recognizing multilingual text embedded in images and translating it into the target language with the support of visual context. While recent Large Vision-Language Models (LVLMs) have demonstrated strong multilingual and visual understanding capabilities, there is a lack of systematic evaluation and understanding of their performance on VLT. In this work, we present a comprehensive study of VLT from three key perspectives: data quality, model architecture, and evaluation metrics. (1) We identify critical limitations in existing datasets, particularly in semantic and cultural fidelity, and introduce AibTrans -- a multilingual, parallel, human-verified dataset with OCR-corrected annotations. (2) We benchmark 11 commercial LVLMs/LLMs and 6 state-of-the-art open-source models across end-to-end and cascaded architectures, revealing their OCR dependency and contrasting generation versus reasoning behaviors. (3) We propose Density-Aware Evaluation to address metric reliability issues under varying contextual complexity, introducing the DA Score as a more robust measure of translation quality. Building upon these findings, we establish a new evaluation benchmark for VLT. Notably, we observe that fine-tuning LVLMs on high-resource language pairs degrades cross-lingual performance, and we propose a balanced multilingual fine-tuning strategy that effectively adapts LVLMs to VLT without sacrificing their generalization ability.