End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification

TL;DR

This paper introduces a novel end-to-end non-autoregressive neural machine translation model based on connectionist temporal classification, enabling faster inference while maintaining translation quality.

Contribution

It presents a new non-autoregressive architecture that can be trained end-to-end, unlike previous multi-step methods, improving translation speed without sacrificing quality.

Findings

Significant speedup over autoregressive models.

Translation quality comparable to other non-autoregressive models.

Effective on WMT English-Romanian and English-German datasets.

Abstract

Autoregressive decoding is the only part of sequence-to-sequence models that prevents them from massive parallelization at inference time. Non-autoregressive models enable the decoder to generate all output symbols independently in parallel. We present a novel non-autoregressive architecture based on connectionist temporal classification and evaluate it on the task of neural machine translation. Unlike other non-autoregressive methods which operate in several steps, our model can be trained end-to-end. We conduct experiments on the WMT English-Romanian and English-German datasets. Our models achieve a significant speedup over the autoregressive models, keeping the translation quality comparable to other non-autoregressive models.