Diformer: Directional Transformer for Neural Machine Translation

TL;DR

Diformer introduces a unified directional transformer model that jointly captures autoregressive and non-autoregressive translation directions, improving performance and latency in neural machine translation.

Contribution

It proposes a novel framework that preserves original training objectives of AR and NAR models by modeling three generation directions with a direction variable.

Findings

Outperforms existing unified models by over 1.5 BLEU points on WMT benchmarks.

Achieves competitive results with state-of-the-art independent AR and NAR models.

Effectively combines AR and NAR advantages in a single model.

Abstract

Autoregressive (AR) and Non-autoregressive (NAR) models have their own superiority on the performance and latency, combining them into one model may take advantage of both. Current combination frameworks focus more on the integration of multiple decoding paradigms with a unified generative model, e.g. Masked Language Model. However, the generalization can be harmful to the performance due to the gap between training objective and inference. In this paper, we aim to close the gap by preserving the original objective of AR and NAR under a unified framework. Specifically, we propose the Directional Transformer (Diformer) by jointly modelling AR and NAR into three generation directions (left-to-right, right-to-left and straight) with a newly introduced direction variable, which works by controlling the prediction of each token to have specific dependencies under that direction. The unification achieved by direction successfully preserves the original dependency assumption used in AR and NAR, retaining both generalization and performance. Experiments on 4 WMT benchmarks demonstrate that Diformer outperforms current united-modelling works with more than 1.5 BLEU points for both AR and NAR decoding, and is also competitive to the state-of-the-art independent AR and NAR models.