Exploring Alignment in Shared Cross-lingual Spaces

TL;DR

This paper investigates how multilingual models align concepts across languages in their latent spaces, introducing metrics to quantify this alignment and analyzing models like mT5, mBERT, and XLM-R across various tasks.

Contribution

It introduces two metrics, extit{CA} and extit{CO}, to measure concept alignment and overlap in multilingual embeddings, and analyzes how these evolve with model depth and fine-tuning.

Findings

Deeper layers show increased cross-lingual alignment.

Fine-tuning enhances alignment in latent space.

Task-specific calibration explains zero-shot capabilities.

Abstract

Despite their remarkable ability to capture linguistic nuances across diverse languages, questions persist regarding the degree of alignment between languages in multilingual embeddings. Drawing inspiration from research on high-dimensional representations in neural language models, we employ clustering to uncover latent concepts within multilingual models. Our analysis focuses on quantifying the \textit{alignment} and \textit{overlap} of these concepts across various languages within the latent space. To this end, we introduce two metrics \CA{} and \CO{} aimed at quantifying these aspects, enabling a deeper exploration of multilingual embeddings. Our study encompasses three multilingual models (\texttt{mT5}, \texttt{mBERT}, and \texttt{XLM-R}) and three downstream tasks (Machine Translation, Named Entity Recognition, and Sentiment Analysis). Key findings from our analysis include: i) deeper layers in the network demonstrate increased cross-lingual \textit{alignment} due to the presence of language-agnostic concepts, ii) fine-tuning of the models enhances \textit{alignment} within the latent space, and iii) such task-specific calibration helps in explaining the emergence of zero-shot capabilities in the models.\footnote{The code is available at \url{https://github.com/baselmousi/multilingual-latent-concepts}}