Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings

TL;DR

This paper introduces an unsupervised optimal transport-based alignment method for fine-tuning multilingual contextualized embeddings, improving cross-lingual transfer without requiring pre-aligned word pairs.

Contribution

It proposes using optimal transport as an alignment objective during fine-tuning, enabling context-aware, unsupervised, and flexible alignment of multilingual embeddings.

Findings

Improved performance on XNLI and XQuAD benchmarks.

Achieved competitive results with state-of-the-art methods.

Demonstrated effectiveness of unsupervised alignment in multilingual settings.

Abstract

Recent studies have proposed different methods to improve multilingual word representations in contextualized settings including techniques that align between source and target embedding spaces. For contextualized embeddings, alignment becomes more complex as we additionally take context into consideration. In this work, we propose using Optimal Transport (OT) as an alignment objective during fine-tuning to further improve multilingual contextualized representations for downstream cross-lingual transfer. This approach does not require word-alignment pairs prior to fine-tuning that may lead to sub-optimal matching and instead learns the word alignments within context in an unsupervised manner. It also allows different types of mappings due to soft matching between source and target sentences. We benchmark our proposed method on two tasks (XNLI and XQuAD) and achieve improvements over baselines as well as competitive results compared to similar recent works.