DOCmT5: Document-Level Pretraining of Multilingual Language Models

TL;DR

DOCmT5 is a multilingual document-level pretrained model utilizing a novel document reordering machine translation objective, significantly improving performance on various document translation and summarization tasks across multiple languages.

Contribution

Introduces DOCmT5 with a new pretraining objective, enhancing document understanding and generation in multilingual settings, achieving state-of-the-art results.

Findings

Significant BLEU score improvements on document translation tasks.

State-of-the-art results on WMT20 De-En and IWSLT15 Zh-En.

Effective combination of monolingual and cross-lingual pretraining enhances performance.

Abstract

In this paper, we introduce DOCmT5, a multilingual sequence-to-sequence language model pretrained with large scale parallel documents. While previous approaches have focused on leveraging sentence-level parallel data, we try to build a general-purpose pretrained model that can understand and generate long documents. We propose a simple and effective pretraining objective - Document reordering Machine Translation (DrMT), in which the input documents that are shuffled and masked need to be translated. DrMT brings consistent improvements over strong baselines on a variety of document-level generation tasks, including over 12 BLEU points for seen-language-pair document-level MT, over 7 BLEU points for unseen-language-pair document-level MT and over 3 ROUGE-1 points for seen-language-pair cross-lingual summarization. We achieve state-of-the-art (SOTA) on WMT20 De-En and IWSLT15 Zh-En document translation tasks. We also conduct extensive analysis on various factors for document pretraining, including (1) The effects of pretraining data quality and (2) The effects of combining mono-lingual and cross-lingual pretraining. We plan to make our model checkpoints publicly available.