Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining

TL;DR

This paper investigates how multilingual language models develop translation abilities during early training, revealing a two-phase process involving copying and the emergence of general translation mechanisms.

Contribution

It introduces a new dataset and fine-grained analysis methods to study the evolution of translation capabilities in multilingual models during pretraining.

Findings

Models quickly learn basic linguistic skills alongside copying.

Translation develops in two phases: initial copying and later generalization.

Translation mechanisms emerge after initial surface-level similarities.

Abstract

Large language models exhibit impressive cross-lingual capabilities. However, prior work analyzes this phenomenon through isolated factors and at sparse points during training, limiting our understanding of how cross-lingual generalization emerges--particularly in the early phases of learning. To study the early trajectory of linguistic and translation capabilities, we pretrain a multilingual 1.7B model on nine diverse languages, capturing checkpoints at a much finer granularity. We further introduce a novel word-level translation dataset and trace how translation develops over training through behavioral analyses, model-component analysis, and parameter-based ablations. We find that the model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which more generalizing translation mechanisms are developed while copying is refined. Together, these findings provide a fine-grained view of how cross-lingual generalization develops during multilingual pretraining.