Multilingual Bidirectional Unsupervised Translation Through Multilingual Finetuning and Back-Translation

TL;DR

This paper introduces EcXTra, a simple yet effective two-stage multilingual translation method that leverages pretrained models, multilingual fine-tuning, and back-translation to improve translation quality for unseen languages, achieving state-of-the-art results.

Contribution

The paper presents a novel two-stage approach combining multilingual fine-tuning and back-translation with pretrained models for zero-shot multilingual translation.

Findings

Achieved state-of-the-art BLEU scores for English-to-Kazakh translation.

Each back-translation round improves bidirectional translation performance.

The method is effective across 7 low-resource languages.

Abstract

We propose a two-stage approach for training a single NMT model to translate unseen languages both to and from English. For the first stage, we initialize an encoder-decoder model to pretrained XLM-R and RoBERTa weights, then perform multilingual fine-tuning on parallel data in 40 languages to English. We find this model can generalize to zero-shot translations on unseen languages. For the second stage, we leverage this generalization ability to generate synthetic parallel data from monolingual datasets, then bidirectionally train with successive rounds of back-translation. Our approach, which we EcXTra (English-centric Crosslingual (X) Transfer), is conceptually simple, only using a standard cross-entropy objective throughout. It is also data-driven, sequentially leveraging auxiliary parallel data and monolingual data. We evaluate unsupervised NMT results for 7 low-resource languages, and find that each round of back-translation training further refines bidirectional performance. Our final single EcXTra-trained model achieves competitive translation performance in all translation directions, notably establishing a new state-of-the-art for English-to-Kazakh (22.9 > 10.4 BLEU). Our code is available at https://github.com/manestay/EcXTra .