Cross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning

TL;DR

This paper explores using visual information from image-caption datasets to align multilingual sentence representations, providing an efficient alternative to bitexts, especially for low-resource languages, and demonstrates their effectiveness in cross-lingual tasks.

Contribution

It introduces a contrastive image-caption tuning method that aligns multilingual text representations via visual data, enabling cross-lingual NLU and retrieval without extensive multilingual training.

Findings

Visual alignment can implicitly align multilingual text representations.

Unseen languages in pretraining can be incorporated post-hoc.

Aligned representations improve cross-lingual NLU and retrieval performance.

Abstract

Multilingual alignment of sentence representations has mostly required bitexts to bridge the gap between languages. We investigate whether visual information can bridge this gap instead. Image caption datasets are very easy to create without requiring multilingual expertise, so this offers a more efficient alternative for low-resource languages. We find that multilingual image-caption alignment can implicitly align the text representations between languages, languages unseen by the encoder in pretraining can be incorporated into this alignment post-hoc, and these aligned representations are usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval.