MVP: Multi-task Supervised Pre-training for Natural Language Generation

TL;DR

This paper introduces MVP, a multi-task supervised pre-training approach for natural language generation that unifies diverse datasets into a text-to-text format, leading to state-of-the-art results on multiple NLG tasks.

Contribution

The paper presents a large-scale supervised pre-training method using a unified dataset and soft prompts, improving NLG performance over existing models.

Findings

Achieves state-of-the-art results on 13 out of 17 datasets.

Outperforms BART by 9.3% and Flan-T5 by 5.8%.

Demonstrates the effectiveness of supervised multi-task pre-training.

Abstract

Pre-trained language models (PLMs) have achieved remarkable success in natural language generation (NLG) tasks. Up to now, most NLG-oriented PLMs are pre-trained in an unsupervised manner using the large-scale general corpus. In the meanwhile, an increasing number of models pre-trained with labeled data (i.e. "supervised pre-training") showcase superior performance compared to unsupervised pre-trained models. Motivated by the success of supervised pre-training, we propose Multi-task superVised Pre-training (MVP) for natural language generation. We collect a large-scale natural language generation corpus, MVPCorpus, from $77$ datasets over $11$ diverse NLG tasks. Then we unify these examples into a general text-to-text format to pre-train the text generation model MVP in a supervised manner. For each task, we further pre-train specific soft prompts to stimulate the model's capacity to perform a specific task. Our MVP model can be seen as a practice that utilizes recent instruction tuning on relatively small PLMs. Extensive experiments have demonstrated the effectiveness and generality of our MVP model in a number of NLG tasks, which achieves state-of-the-art performance on $13$ out of $17$ datasets, outperforming BART by $9.3\%$ and Flan-T5 by $5.8\%$.