MODS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collections
Nishant Balepur, Alexa Siu, Nedim Lipka, Franck Dernoncourt, Tong Sun,, Jordan Boyd-Graber, Puneet Mathur

TL;DR
This paper introduces MODS, a multi-LLM framework that generates balanced, comprehensive summaries for debatable queries by mimicking human panel discussions, addressing limitations of previous query-focused summarization methods.
Contribution
MODS is a novel multi-LLM approach that models document summarization as a moderated discussion, improving coverage and balance in debatable query summaries.
Findings
MODS outperforms SOTA by 38-59% in coverage and balance metrics.
MODS produces more balanced and readable summaries according to user evaluations.
Experiments on ConflictingQA and DebateQFS datasets demonstrate effectiveness.
Abstract
Query-focused summarization (QFS) gives a summary of documents to answer a query. Past QFS work assumes queries have one answer, ignoring debatable ones (Is law school worth it?). We introduce Debatable QFS (DQFS), a task to create summaries that answer debatable queries via documents with opposing perspectives; summaries must comprehensively cover all sources and balance perspectives, favoring no side. These goals elude LLM QFS systems, which: 1) lack structured content plans, failing to guide LLMs to write balanced summaries, and 2) use the same query to retrieve contexts across documents, failing to cover all perspectives specific to each document's content. To overcome this, we design MODS, a multi-LLM framework mirroring human panel discussions. MODS treats documents as individual Speaker LLMs and has a Moderator LLM that picks speakers to respond to tailored queries for planned…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsTopic Modeling · Music and Audio Processing · Natural Language Processing Techniques
