Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation

TL;DR

This study investigates the effectiveness of multi-encoder neural machine translation models, revealing that noise generation and dropout during training significantly contribute to performance improvements, especially with limited data.

Contribution

The paper demonstrates that noise and dropout are key factors in multi-encoder NMT, challenging the assumption that multiple encoders inherently improve translation quality.

Findings

Noise acts as a regularizer, improving robustness.

Careful noise and dropout tuning achieves state-of-the-art results.

Multi-encoder benefits are partly due to noise-induced robustness.

Abstract

In encoder-decoder neural models, multiple encoders are in general used to represent the contextual information in addition to the individual sentence. In this paper, we investigate multi-encoder approaches in documentlevel neural machine translation (NMT). Surprisingly, we find that the context encoder does not only encode the surrounding sentences but also behaves as a noise generator. This makes us rethink the real benefits of multi-encoder in context-aware translation - some of the improvements come from robust training. We compare several methods that introduce noise and/or well-tuned dropout setup into the training of these encoders. Experimental results show that noisy training plays an important role in multi-encoder-based NMT, especially when the training data is small. Also, we establish a new state-of-the-art on IWSLT Fr-En task by careful use of noise generation and dropout methods.