Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model

TL;DR

This paper introduces new attention-based encoder-decoder models with implicit distortion and fertility mechanisms to improve translation quality in neural machine translation, addressing alignment issues.

Contribution

The paper proposes novel variations of attention mechanisms incorporating implicit distortion and fertility models, enhancing translation accuracy over standard models.

Findings

Achieved a 2 BLEU point improvement over original models.

Demonstrated that addressing alignment issues improves translation quality.

Compared new models with existing approaches on machine translation tasks.

Abstract

Neural machine translation has shown very promising results lately. Most NMT models follow the encoder-decoder framework. To make encoder-decoder models more flexible, attention mechanism was introduced to machine translation and also other tasks like speech recognition and image captioning. We observe that the quality of translation by attention-based encoder-decoder can be significantly damaged when the alignment is incorrect. We attribute these problems to the lack of distortion and fertility models. Aiming to resolve these problems, we propose new variations of attention-based encoder-decoder and compare them with other models on machine translation. Our proposed method achieved an improvement of 2 BLEU points over the original attention-based encoder-decoder.