Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining

TL;DR

This paper demonstrates that bilingual alignment techniques can be transferred to improve multilingual sentence representations, enabling effective unsupervised bitext mining across multiple languages.

Contribution

It introduces dual-pivot transfer, showing that models trained on one language pair can generalize to others for multilingual alignment.

Findings

Unsupervised models achieve state-of-the-art unsupervised retrieval performance.

Single-pair supervised models approach multilingually supervised model performance.

Bilingual training techniques effectively produce multilingual sentence representations.

Abstract

This work presents methods for learning cross-lingual sentence representations using paired or unpaired bilingual texts. We hypothesize that the cross-lingual alignment strategy is transferable, and therefore a model trained to align only two languages can encode multilingually more aligned representations. We thus introduce dual-pivot transfer: training on one language pair and evaluating on other pairs. To study this theory, we design unsupervised models trained on unpaired sentences and single-pair supervised models trained on bitexts, both based on the unsupervised language model XLM-R with its parameters frozen. The experiments evaluate the models as universal sentence encoders on the task of unsupervised bitext mining on two datasets, where the unsupervised model reaches the state of the art of unsupervised retrieval, and the alternative single-pair supervised model approaches the performance of multilingually supervised models. The results suggest that bilingual training techniques as proposed can be applied to get sentence representations with multilingual alignment.