Modeling Coverage for Non-Autoregressive Neural Machine Translation

TL;DR

This paper introduces a novel coverage modeling approach for Non-Autoregressive Neural Machine Translation to reduce translation errors and improve quality by tracking token coverage during translation.

Contribution

It proposes a coverage-based NAT model with token-level refinement and sentence-level agreement, addressing over- and under-translation issues.

Findings

Significant reduction in translation errors.

Improved translation quality on WMT datasets.

Outperforms baseline NAT systems.

Abstract

Non-Autoregressive Neural Machine Translation (NAT) has achieved significant inference speedup by generating all tokens simultaneously. Despite its high efficiency, NAT usually suffers from two kinds of translation errors: over-translation (e.g. repeated tokens) and under-translation (e.g. missing translations), which eventually limits the translation quality. In this paper, we argue that these issues of NAT can be addressed through coverage modeling, which has been proved to be useful in autoregressive decoding. We propose a novel Coverage-NAT to model the coverage information directly by a token-level coverage iterative refinement mechanism and a sentence-level coverage agreement, which can remind the model if a source token has been translated or not and improve the semantics consistency between the translation and the source, respectively. Experimental results on WMT14 En-De and WMT16 En-Ro translation tasks show that our method can alleviate those errors and achieve strong improvements over the baseline system.