Factual Dialogue Summarization via Learning from Large Language Models

TL;DR

This paper proposes a symbolic knowledge distillation method using contrastive learning to improve factual consistency in dialogue summarization models, especially smaller ones, by leveraging large language models.

Contribution

It introduces a novel approach combining zero-shot knowledge extraction and contrastive learning to enhance factual accuracy in smaller dialogue summarization models.

Findings

Outperforms strong baselines in factual consistency.

Maintains coherence, fluency, and relevance.

Effective across multiple pretrained models.

Abstract

Factual consistency is an important quality in dialogue summarization. Large language model (LLM)-based automatic text summarization models generate more factually consistent summaries compared to those by smaller pretrained language models, but they face deployment challenges in real-world applications due to privacy or resource constraints. In this paper, we investigate the use of symbolic knowledge distillation to improve the factual consistency of smaller pretrained models for dialogue summarization. We employ zero-shot learning to extract symbolic knowledge from LLMs, generating both factually consistent (positive) and inconsistent (negative) summaries. We then apply two contrastive learning objectives on these summaries to enhance smaller summarization models. Experiments with BART, PEGASUS, and Flan-T5 indicate that our approach surpasses strong baselines that rely on complex data augmentation strategies. Our approach achieves better factual consistency while maintaining coherence, fluency, and relevance, as confirmed by various automatic evaluation metrics. We also provide access to the data and code to facilitate future research.