ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation

TL;DR

ELMER is a pre-trained language model designed for non-autoregressive text generation, achieving high efficiency and quality by modeling token dependencies and using an innovative pre-training method, significantly improving inference speed and performance.

Contribution

ELMER introduces explicit token dependency modeling and a novel Layer Permutation Language Modeling pre-training objective for non-autoregressive text generation.

Findings

ELMER outperforms existing NAR models in quality.

ELMER narrows the performance gap with AR models.

ELMER achieves over 10 times faster inference speed.

Abstract

We study the text generation task under the approach of pre-trained language models (PLMs). Typically, an auto-regressive (AR) method is adopted for generating texts in a token-by-token manner. Despite many advantages of AR generation, it usually suffers from inefficient inference. Therefore, non-autoregressive (NAR) models are proposed to generate all target tokens simultaneously. However, NAR models usually generate texts of lower quality due to the absence of token dependency in the output text. In this paper, we propose ELMER: an efficient and effective PLM for NAR text generation to explicitly model the token dependency during NAR generation. By leveraging the early exit technique, ELMER enables the token generations at different layers, according to their prediction confidence (a more confident token will exit at a lower layer). Besides, we propose a novel pre-training objective, Layer Permutation Language Modeling, to pre-train ELMER by permuting the exit layer for each token in sequences. Experiments on three text generation tasks show that ELMER significantly outperforms NAR models and further narrows the performance gap with AR PLMs (\eg ELMER (29.92) vs BART (30.61) ROUGE-L in XSUM) while achieving over 10 times inference speedup.