On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation

TL;DR

This paper investigates how pre-training and back-translation complement each other in neural machine translation, showing that their combination yields state-of-the-art results by leveraging their distinct contributions to encoder and decoder improvements.

Contribution

It is the first study to analyze the complementarity between pre-training and back-translation in NMT, demonstrating their combined effectiveness and proposing enhancements like Tagged BT.

Findings

Pre-training mainly benefits the encoder module.

Back-translation enhances the decoder performance.

Combining PT and BT achieves state-of-the-art results on benchmarks.

Abstract

Pre-training (PT) and back-translation (BT) are two simple and powerful methods to utilize monolingual data for improving the model performance of neural machine translation (NMT). This paper takes the first step to investigate the complementarity between PT and BT. We introduce two probing tasks for PT and BT respectively and find that PT mainly contributes to the encoder module while BT brings more benefits to the decoder. Experimental results show that PT and BT are nicely complementary to each other, establishing state-of-the-art performances on the WMT16 English-Romanian and English-Russian benchmarks. Through extensive analyses on sentence originality and word frequency, we also demonstrate that combining Tagged BT with PT is more helpful to their complementarity, leading to better translation quality. Source code is freely available at https://github.com/SunbowLiu/PTvsBT.