Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment

TL;DR

This paper demonstrates that using multi-way parallel corpora for contrastive learning significantly enhances multilingual and cross-lingual representations, leading to improved performance on various natural language understanding tasks across multiple languages.

Contribution

The authors introduce a novel approach of leveraging multi-way parallel texts for contrastive training to improve cross-lingual alignment in pretrained models, surpassing traditional bilingual methods.

Findings

21.3% improvement in bitext mining

5.3% gain in semantic similarity

28.4% increase in classification accuracy

Abstract

Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus in a diverse pool of languages can substantially improve multilingual and cross-lingual representations for NLU tasks. We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages and achieve strong cross-lingual alignment through contrastive learning. This leads to substantial performance gains across both seen and unseen languages for multiple tasks from the MTEB benchmark evaluated for XLM-Roberta and multilingual BERT base models. Using a multi-way parallel corpus for contrastive training yields substantial gains on bitext mining (21.3%), semantic similarity (5.3%), and classification (28.4%) compared to English-centric (En-X) bilingually parallel data, where X is sampled from a pool of multiple target languages. Furthermore, finetuning mE5 model on a small dataset with multi-way parallelism significantly improves bitext mining compared to one without, underscoring the importance of multi-way cross-lingual supervision even for models already pretrained for high-quality sentence embeddings.