EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Start

TL;DR

EdiT5 is a semi-autoregressive text-editing model that combines non-autoregressive tagging and reordering with autoregressive insertion, achieving faster inference and competitive performance on various NLG tasks.

Contribution

It introduces a novel semi-autoregressive approach that decomposes text editing into tagging, reordering, and insertion, improving speed and flexibility over traditional seq2seq models.

Findings

Up to 25x speedup compared to seq2seq models.

Comparable performance to T5 in high-resource settings.

Outperforms T5 in low-resource scenarios.

Abstract

We present EdiT5 - a novel semi-autoregressive text-editing model designed to combine the strengths of non-autoregressive text-editing and autoregressive decoding. EdiT5 is faster during inference than conventional sequence-to-sequence (seq2seq) models, while being capable of modelling flexible input-output transformations. This is achieved by decomposing the generation process into three sub-tasks: (1) tagging to decide on the subset of input tokens to be preserved in the output, (2) re-ordering to define their order in the output text, and (3) insertion to infill the missing tokens that are not present in the input. The tagging and re-ordering steps, which are responsible for generating the largest portion of the output, are non-autoregressive, while the insertion step uses an autoregressive decoder. Depending on the task, EdiT5 on average requires significantly fewer autoregressive steps, demonstrating speedups of up to 25x when compared to seq2seq models. Quality-wise, EdiT5 is initialized with a pre-trained T5 checkpoint yielding comparable performance to T5 in high-resource settings when evaluated on three NLG tasks: Sentence Fusion, Grammatical Error Correction, and Decontextualization while clearly outperforming T5 in low-resource settings.