Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting

TL;DR

This paper introduces Sequence Span Rewriting (SSR), a novel self-supervised pre-training method for seq2seq models that enhances learning by rewriting imperfect spans, leading to improved performance especially for smaller models.

Contribution

The paper proposes SSR as a new pre-training objective that provides finer learning signals and better aligns with downstream seq2seq tasks, especially benefiting small models.

Findings

SSR significantly improves seq2seq pre-training performance.

SSR is particularly effective for small models with a strong span generator.

SSR offers a new way to transfer knowledge from large to small models.

Abstract

In this paper, we generalize text infilling (e.g., masked language models) by proposing Sequence Span Rewriting (SSR) as a self-supervised sequence-to-sequence (seq2seq) pre-training objective. SSR provides more fine-grained learning signals for text representations by supervising the model to rewrite imperfect spans to ground truth, and it is more consistent than text infilling with many downstream seq2seq tasks that rewrite a source sentences into a target sentence. Our experiments with T5 models on various seq2seq tasks show that SSR can substantially improve seq2seq pre-training. Moreover, we observe SSR is especially helpful to improve pre-training a small-size seq2seq model with a powerful imperfect span generator, which indicates a new perspective of transferring knowledge from a large model to a smaller model for seq2seq pre-training.