Memory-enhanced Decoder for Neural Machine Translation

TL;DR

This paper introduces MemDec, an external memory-augmented RNN decoder for neural machine translation, which improves translation quality by better capturing decoding information through content-based memory read/write operations.

Contribution

The paper presents MemDec, a novel memory-enhanced RNN decoder that improves translation performance by integrating a fixed-size external memory with content-based addressing.

Findings

Achieved 4.8 BLEU improvement on Groundhog.

Achieved 5.3 BLEU improvement on Moses.

Demonstrated state-of-the-art results with the same training data.

Abstract

We propose to enhance the RNN decoder in a neural machine translator (NMT) with external memory, as a natural but powerful extension to the state in the decoding RNN. This memory-enhanced RNN decoder is called \textsc{MemDec}. At each time during decoding, \textsc{MemDec} will read from this memory and write to this memory once, both with content-based addressing. Unlike the unbounded memory in previous work\cite{RNNsearch} to store the representation of source sentence, the memory in \textsc{MemDec} is a matrix with pre-determined size designed to better capture the information important for the decoding process at each time step. Our empirical study on Chinese-English translation shows that it can improve by $4.8$ BLEU upon Groundhog and $5.3$ BLEU upon on Moses, yielding the best performance achieved with the same training set.