Faithful Summarisation under Disagreement via Belief-Level Aggregation

TL;DR

This paper proposes a belief-level aggregation approach for summarisation that explicitly models disagreement, improving faithfulness in opinion-heavy multi-document summarisation compared to traditional LLM-based methods.

Contribution

It introduces a novel disagreement-aware synthesis pipeline that separates belief aggregation from language generation, enhancing faithfulness in summarisation.

Findings

Belief-level aggregation with explicit conflict modeling improves summary faithfulness.

Large models can match belief aggregation when integrated during generation, but are less stable.

Simple prompting with belief aggregation yields consistent performance across models.

Abstract

Opinion and multi-document summarisation often involve genuinely conflicting viewpoints, yet many existing approaches, particularly LLM-based systems, implicitly smooth disagreement and over-represent majority opinions. This limits the faithfulness of generated summaries in opinion-heavy settings. We introduce a disagreement-aware synthesis pipeline that separates belief-level aggregation from language generation. Documents are first represented as structured belief sets and aggregated using distance-based belief merging operators that explicitly model conflict. Large language models are then used only to realise the aggregated beliefs as natural language summaries. We evaluate the approach across multiple model families and scales, comparing it to methods that perform explicit aggregation during generation. Our results show that while sufficiently large models can match belief-level aggregation when aggregation is handled at generation time, this behaviour is not stable across architectures or capacities. In contrast, belief-level aggregation combined with simple prompting yields consistently strong disagreement-aware performance across models, while maintaining fluent and grounded summaries.