Towards Inducing Long-Context Abilities in Multilingual Neural Machine Translation Models

TL;DR

This paper explores transitioning pre-trained multilingual NMT models from sinusoidal to relative positional embeddings using parameter-efficient fine-tuning, enhancing long-context translation abilities without performance loss.

Contribution

It demonstrates effective methods for switching positional embeddings in multilingual NMT models and inducing long-context capabilities with minimal additional data.

Findings

Relative PEs outperform sinusoidal PEs on document-level benchmarks.

RoPE consistently outperforms ALiBi and sinusoidal embeddings.

A small amount of long-context data enables cross-lingual length generalization.

Abstract

Neural Machine Translation (NMT) models have traditionally used Sinusoidal Positional Embeddings (PEs), which often struggle to capture long-range dependencies and are inefficient for handling extended context or document-level translation tasks. This work addresses the challenge of transitioning pre-trained NMT models from absolute Sinusoidal PEs to Relative PEs, such as RoPE and ALiBi, without compromising performance. We demonstrate that parameter-efficient fine-tuning, using only a small amount of high-quality data, can successfully facilitate this transition. Experimental results indicate that switching from Sinusoidal to Relative PEs results in competitive translation quality on sentence-level evaluation benchmarks. Additionally, models trained with RoPE consistently outperform those using ALiBi and Sinusoidal PEs on document-level benchmarks across both string-based metrics and qualitative evaluations. Moreover, we find that a small amount of long-context data in a few languages is sufficient for cross-lingual length generalization, thereby inducing long-context capabilities.