Improving Neural Machine Translation with Pre-trained Representation

TL;DR

This paper introduces a novel method to enhance neural machine translation by leveraging sentence-level contextual representations learned from monolingual data, leading to improved translation quality especially in low-resource scenarios.

Contribution

The paper proposes a new structure for acquiring and integrating sentence-level contextual representations from monolingual data into NMT models, which was not fully exploited before.

Findings

Improves translation quality on Chinese-English and German-English tasks.

Effective in low-resource English-Turkish translation.

Outperforms strong Transformer baselines.

Abstract

Monolingual data has been demonstrated to be helpful in improving the translation quality of neural machine translation (NMT). The current methods stay at the usage of word-level knowledge, such as generating synthetic parallel data or extracting information from word embedding. In contrast, the power of sentence-level contextual knowledge which is more complex and diverse, playing an important role in natural language generation, has not been fully exploited. In this paper, we propose a novel structure which could leverage monolingual data to acquire sentence-level contextual representations. Then, we design a framework for integrating both source and target sentence-level representations into NMT model to improve the translation quality. Experimental results on Chinese-English, German-English machine translation tasks show that our proposed model achieves improvement over strong Transformer baselines, while experiments on English-Turkish further demonstrate the effectiveness of our approach in the low-resource scenario.