Globetrotter: Connecting Languages by Connecting Images

TL;DR

This paper presents Globetrotter, a novel approach that leverages visual data to connect multiple languages for translation, bypassing the need for parallel corpora by aligning text segments based on associated images.

Contribution

The paper introduces a visual-based method for multilingual translation that does not require parallel data, trained on a new multi-language dataset with images, outperforming previous unsupervised translation methods.

Findings

Outperforms previous unsupervised translation methods

Effective in aligning multilingual text segments via images

Trained on a new dataset of 50+ languages with images

Abstract

Machine translation between many languages at once is highly challenging, since training with ground truth requires supervision between all language pairs, which is difficult to obtain. Our key insight is that, while languages may vary drastically, the underlying visual appearance of the world remains consistent. We introduce a method that uses visual observations to bridge the gap between languages, rather than relying on parallel corpora or topological properties of the representations. We train a model that aligns segments of text from different languages if and only if the images associated with them are similar and each image in turn is well-aligned with its textual description. We train our model from scratch on a new dataset of text in over fifty languages with accompanying images. Experiments show that our method outperforms previous work on unsupervised word and sentence translation using retrieval. Code, models and data are available on globetrotter.cs.columbia.edu.