Cross-lingual Visual Pre-training for Multimodal Machine Translation

TL;DR

This paper introduces a novel cross-lingual visual pre-training method that combines language and visual data to improve multimodal machine translation, achieving state-of-the-art results and providing insights into grounded representations.

Contribution

It extends translation language modeling with visual grounding and pre-trains on three-way parallel vision-language data, advancing multimodal translation performance.

Findings

Achieves state-of-the-art results in multimodal machine translation

Demonstrates the effectiveness of visually-grounded cross-lingual representations

Provides qualitative insights into learned grounded representations

Abstract

Pre-trained language models have been shown to improve performance in many natural language tasks substantially. Although the early focus of such models was single language pre-training, recent advances have resulted in cross-lingual and visual pre-training methods. In this paper, we combine these two approaches to learn visually-grounded cross-lingual representations. Specifically, we extend the translation language modelling (Lample and Conneau, 2019) with masked region classification and perform pre-training with three-way parallel vision & language corpora. We show that when fine-tuned for multimodal machine translation, these models obtain state-of-the-art performance. We also provide qualitative insights into the usefulness of the learned grounded representations.