Non-autoregressive Transformer by Position Learning

TL;DR

This paper introduces PNAT, a non-autoregressive transformer that explicitly models word positions as latent variables, significantly improving performance on machine translation and paraphrase generation tasks.

Contribution

It proposes a novel position modeling approach in non-autoregressive transformers, enhancing generation quality by explicitly incorporating position information.

Findings

PNAT achieves top results on translation tasks

Outperforms several strong baselines

Explicit position modeling improves text generation quality

Abstract

Non-autoregressive models are promising on various text generation tasks. Previous work hardly considers to explicitly model the positions of generated words. However, position modeling is an essential problem in non-autoregressive text generation. In this study, we propose PNAT, which incorporates positions as a latent variable into the text generative process. Experimental results show that PNAT achieves top results on machine translation and paraphrase generation tasks, outperforming several strong baselines.