Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarization

TL;DR

This paper introduces a lightweight, domain-oriented prefix-tuning method for zero-shot dialogue summarization, improving domain adaptation and generalization without extensive pre-training.

Contribution

It proposes a novel prefix-tuning approach with domain-specific initialization and discrete prompts to enhance zero-shot domain adaptation in dialogue summarization.

Findings

Effective in zero-shot domain adaptation

Improves generalization across multiple dialogue domains

Validated on TODSum and QMSum datasets

Abstract

The most advanced abstractive dialogue summarizers lack generalization ability on new domains and the existing researches for domain adaptation in summarization generally rely on large-scale pre-trainings. To explore the lightweight fine-tuning methods for domain adaptation of dialogue summarization, in this paper, we propose an efficient and generalizable Domain-Oriented Prefix-tuning model, which utilizes a domain word initialized prefix module to alleviate domain entanglement and adopts discrete prompts to guide the model to focus on key contents of dialogues and enhance model generalization. We conduct zero-shot experiments and build domain adaptation benchmarks on two multi-domain dialogue summarization datasets, TODSum and QMSum. Adequate experiments and qualitative analysis prove the effectiveness of our methods.