Sparse and Constrained Attention for Neural Machine Translation

TL;DR

This paper introduces a new sparse and constrained attention mechanism for neural machine translation that improves coverage by controlling how source words are attended to, leading to more accurate translations.

Contribution

It proposes a novel constrained sparsemax attention transformation that is differentiable and sparse, addressing coverage issues in NMT.

Findings

Constrained sparsemax improves translation coverage.

The method is effective across three language pairs.

It outperforms previous attention mechanisms in coverage metrics.

Abstract

In NMT, words are sometimes dropped from the source or generated repeatedly in the translation. We explore novel strategies to address the coverage problem that change only the attention transformation. Our approach allocates fertilities to source words, used to bound the attention each word can receive. We experiment with various sparse and constrained attention transformations and propose a new one, constrained sparsemax, shown to be differentiable and sparse. Empirical evaluation is provided in three languages pairs.