Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment

TL;DR

This paper proposes a modular training approach for multilingual sentence encoders that separates language-specific monolingual modules from cross-lingual alignment adapters, improving performance and balance across tasks and languages.

Contribution

It introduces a modular training framework that mitigates multilinguality issues by decoupling monolingual and cross-lingual training, especially benefiting low-resource languages.

Findings

Improved performance on semantic similarity and relatedness tasks.

Enhanced zero-shot transfer capabilities for sentence classification.

Better balance across monolingual and cross-lingual tasks.

Abstract

Multilingual sentence encoders (MSEs) are commonly obtained by training multilingual language models to map sentences from different languages into a shared semantic space. As such, they are subject to curse of multilinguality, a loss of monolingual representational accuracy due to parameter sharing. Another limitation of MSEs is the trade-off between different task performance: cross-lingual alignment training distorts the optimal monolingual structure of semantic spaces of individual languages, harming the utility of sentence embeddings in monolingual tasks; cross-lingual tasks, such as cross-lingual semantic similarity and zero-shot transfer for sentence classification, may also require conflicting cross-lingual alignment strategies. In this work, we address both issues by means of modular training of sentence encoders. We first train language-specific monolingual modules to mitigate negative interference between languages (i.e., the curse). We then align all non-English sentence embeddings to the English by training cross-lingual alignment adapters, preventing interference with monolingual specialization from the first step. We train the cross-lingual adapters with two different types of data to resolve the conflicting requirements of different cross-lingual tasks. Monolingual and cross-lingual results on semantic text similarity and relatedness, bitext mining and sentence classification show that our modular solution achieves better and more balanced performance across all the tasks compared to full-parameter training of monolithic multilingual sentence encoders, especially benefiting low-resource languages.