Cross-lingual Language Model Pretraining

TL;DR

This paper introduces cross-lingual pretraining methods for language models, achieving state-of-the-art results in cross-lingual tasks, machine translation, and classification across multiple languages.

Contribution

It proposes unsupervised and supervised cross-lingual pretraining techniques, extending generative pretraining to multiple languages with new objectives.

Findings

Achieved 4.9% accuracy improvement on XNLI

Surpassed previous BLEU scores in machine translation tasks

Set new state-of-the-art BLEU scores for Romanian-English translation

Abstract

Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.