Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization

TL;DR

This paper introduces a multi-stage pre-training approach enhanced by ChatGPT for multi-scenario multi-domain dialogue summarization, significantly improving adaptability and performance across diverse datasets and settings.

Contribution

It proposes a novel multi-stage pre-training strategy tailored for multi-scenario multi-domain dialogue summarization, utilizing large-scale data and ChatGPT-annotated summaries.

Findings

Outperforms previous models in full fine-tuning, zero-shot, and few-shot scenarios.

Effective domain-aware and task-oriented pre-training enhances model adaptability.

Achieves significant improvements across multiple dialogue summarization datasets.

Abstract

Dialogue summarization involves a wide range of scenarios and domains. However, existing methods generally only apply to specific scenarios or domains. In this study, we propose a new pre-trained model specifically designed for multi-scenario multi-domain dialogue summarization. It adopts a multi-stage pre-training strategy to reduce the gap between the pre-training objective and fine-tuning objective. Specifically, we first conduct domain-aware pre-training using large-scale multi-scenario multi-domain dialogue data to enhance the adaptability of our pre-trained model. Then, we conduct task-oriented pre-training using large-scale multi-scenario multi-domain "dialogue-summary" parallel data annotated by ChatGPT to enhance the dialogue summarization ability of our pre-trained model. Experimental results on three dialogue summarization datasets from different scenarios and domains indicate that our pre-trained model significantly outperforms previous state-of-the-art models in full fine-tuning, zero-shot, and few-shot settings.