Incorporating a Local Translation Mechanism into Non-autoregressive Translation

TL;DR

This paper introduces a local autoregressive translation mechanism into non-autoregressive models, improving translation quality and speed by capturing local dependencies and reducing repetitions.

Contribution

It proposes a novel local autoregressive translation mechanism integrated into NAT models, enhancing performance and efficiency over existing methods.

Findings

Achieves comparable or better accuracy than CMLM with fewer decoding iterations.

Provides a 2.5x speedup in decoding.

Reduces repeated translations and improves performance on longer sentences.

Abstract

In this work, we introduce a novel local autoregressive translation (LAT) mechanism into non-autoregressive translation (NAT) models so as to capture local dependencies among tar-get outputs. Specifically, for each target decoding position, instead of only one token, we predict a short sequence of tokens in an autoregressive way. We further design an efficient merging algorithm to align and merge the out-put pieces into one final output sequence. We integrate LAT into the conditional masked language model (CMLM; Ghazvininejad et al.,2019) and similarly adopt iterative decoding. Empirical results on five translation tasks show that compared with CMLM, our method achieves comparable or better performance with fewer decoding iterations, bringing a 2.5xspeedup. Further analysis indicates that our method reduces repeated translations and performs better at longer sentences.