Temporal Attention Model for Neural Machine Translation

TL;DR

This paper introduces a temporal attention mechanism for neural machine translation that memorizes alignments over time, leading to improved translation quality and robustness compared to existing models.

Contribution

The paper proposes a novel temporal memory-based attention mechanism that enhances NMT by addressing attention deficiency issues.

Findings

Achieves better translation quality over baseline models

Demonstrates robustness across different language pairs

Outperforms strong SMT baselines in some settings

Abstract

Attention-based Neural Machine Translation (NMT) models suffer from attention deficiency issues as has been observed in recent research. We propose a novel mechanism to address some of these limitations and improve the NMT attention. Specifically, our approach memorizes the alignments temporally (within each sentence) and modulates the attention with the accumulated temporal memory, as the decoder generates the candidate translation. We compare our approach against the baseline NMT model and two other related approaches that address this issue either explicitly or implicitly. Large-scale experiments on two language pairs show that our approach achieves better and robust gains over the baseline and related NMT approaches. Our model further outperforms strong SMT baselines in some settings even without using ensembles.