Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement

TL;DR

This paper introduces a deterministic non-autoregressive sequence model that uses iterative refinement, enabling faster decoding in sequence tasks like translation and image captioning without sacrificing quality.

Contribution

It presents a novel non-autoregressive model based on latent variables and denoising autoencoders, applicable across various sequence generation tasks.

Findings

Significantly faster decoding speeds.

Comparable quality to autoregressive models.

Effective on machine translation and image captioning.

Abstract

We propose a conditional non-autoregressive neural sequence model based on iterative refinement. The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task. We extensively evaluate the proposed model on machine translation (En-De and En-Ro) and image caption generation, and observe that it significantly speeds up decoding while maintaining the generation quality comparable to the autoregressive counterpart.