What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarization

TL;DR

This paper introduces a new task called Knowledge-Grounded Discussion Summarization (KGDS), creating a benchmark and evaluation framework to improve summaries that include background context and clarified opinions in discussions.

Contribution

It proposes the first KGDS benchmark with expert annotations, a hierarchical evaluation framework, and evaluates 12 large language models on this challenging task.

Findings

Models often miss key facts in background summaries.

Models struggle to resolve implicit references in opinion summaries.

KGDS remains a difficult task for current large language models.

Abstract

Traditional dialogue summarization primarily focuses on dialogue content, assuming it comprises adequate information for a clear summary. However, this assumption often fails for discussions grounded in shared background, where participants frequently omit context and use implicit references. This results in summaries that are confusing to readers unfamiliar with the background. To address this, we introduce Knowledge-Grounded Discussion Summarization (KGDS), a novel task that produces a supplementary background summary for context and a clear opinion summary with clarified references. To facilitate research, we construct the first KGDS benchmark, featuring news-discussion pairs and expert-created multi-granularity gold annotations for evaluating sub-summaries. We also propose a novel hierarchical evaluation framework with fine-grained and interpretable metrics. Our extensive evaluation of 12 advanced large language models (LLMs) reveals that KGDS remains a significant challenge. The models frequently miss key facts and retain irrelevant ones in background summarization, and often fail to resolve implicit references in opinion summary integration.