Denoising based Sequence-to-Sequence Pre-training for Text Generation

TL;DR

This paper introduces PoDA, a novel seq2seq pre-training method that jointly trains encoder and decoder via denoising, improving text generation tasks like summarization and error correction without task-specific tuning.

Contribution

PoDA is the first to jointly pre-train encoder and decoder with denoising for seq2seq models, maintaining architecture during fine-tuning and enhancing performance.

Findings

Outperforms strong baselines on multiple datasets

Speeds up convergence in text generation tasks

Effective for summarization and grammatical error correction

Abstract

This paper presents a new sequence-to-sequence (seq2seq) pre-training method PoDA (Pre-training of Denoising Autoencoders), which learns representations suitable for text generation tasks. Unlike encoder-only (e.g., BERT) or decoder-only (e.g., OpenAI GPT) pre-training approaches, PoDA jointly pre-trains both the encoder and decoder by denoising the noise-corrupted text, and it also has the advantage of keeping the network architecture unchanged in the subsequent fine-tuning stage. Meanwhile, we design a hybrid model of Transformer and pointer-generator networks as the backbone architecture for PoDA. We conduct experiments on two text generation tasks: abstractive summarization, and grammatical error correction. Results on four datasets show that PoDA can improve model performance over strong baselines without using any task-specific techniques and significantly speed up convergence.