Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining

TL;DR

This paper introduces an unsupervised approach to generate multilingual sentence embeddings using only monolingual data, enabling effective parallel corpus mining especially for low-resource languages.

Contribution

It presents a novel unsupervised method that leverages synthetic data and fine-tunes a pretrained model to produce high-quality multilingual sentence embeddings.

Findings

Up to 22 F1 points improvement over vanilla XLM in corpus mining tasks

Synthetic bilingual corpus enhances results across multiple language pairs

Method enables low-resource languages to benefit from multilingual embeddings

Abstract

Existing models of multilingual sentence embeddings require large parallel data resources which are not available for low-resource languages. We propose a novel unsupervised method to derive multilingual sentence embeddings relying only on monolingual data. We first produce a synthetic parallel corpus using unsupervised machine translation, and use it to fine-tune a pretrained cross-lingual masked language model (XLM) to derive the multilingual sentence representations. The quality of the representations is evaluated on two parallel corpus mining tasks with improvements of up to 22 F1 points over vanilla XLM. In addition, we observe that a single synthetic bilingual corpus is able to improve results for other language pairs.