Probing the Emergence of Cross-lingual Alignment during LLM Training

TL;DR

This paper investigates how cross-lingual alignment naturally emerges during the training of multilingual LLMs, revealing a strong correlation between neuron overlap and transfer performance, and identifying phases of alignment degradation.

Contribution

It introduces intrinsic probing techniques to analyze neuron-level cross-lingual alignment during LLM pre-training, providing new insights into multilingual training dynamics.

Findings

Neuron overlap correlates with zero-shot transfer performance.

Alignment degrades at certain training phases.

Higher model scales show better cross-lingual alignment.

Abstract

Multilingual Large Language Models (LLMs) achieve remarkable levels of zero-shot cross-lingual transfer performance. We speculate that this is predicated on their ability to align languages without explicit supervision from parallel sentences. While representations of translationally equivalent sentences in different languages are known to be similar after convergence, however, it remains unclear how such cross-lingual alignment emerges during pre-training of LLMs. Our study leverages intrinsic probing techniques, which identify which subsets of neurons encode linguistic features, to correlate the degree of cross-lingual neuron overlap with the zero-shot cross-lingual transfer performance for a given model. In particular, we rely on checkpoints of BLOOM, a multilingual autoregressive LLM, across different training steps and model scales. We observe a high correlation between neuron overlap and downstream performance, which supports our hypothesis on the conditions leading to effective cross-lingual transfer. Interestingly, we also detect a degradation of both implicit alignment and multilingual abilities in certain phases of the pre-training process, providing new insights into the multilingual pretraining dynamics.