Chain of Summaries: Summarization Through Iterative Questioning

TL;DR

The paper introduces Chain of Summaries (CoS), an iterative summarization method inspired by dialectical reasoning, which produces dense, general-purpose summaries that improve LLM performance on various datasets and are more accessible for web content use.

Contribution

It presents a novel iterative summarization approach, CoS, that refines summaries through questioning, outperforming existing methods and enhancing LLM understanding and retrieval efficiency.

Findings

CoS outperforms zero-shot LLM baselines by up to 66%.

CoS surpasses specialized summarization methods by up to 27%.

Summaries generated by CoS improve Q&A performance and reduce token usage.

Abstract

Large Language Models (LLMs) are increasingly using external web content. However, much of this content is not easily digestible by LLMs due to LLM-unfriendly formats and limitations of context length. To address this issue, we propose a method for generating general-purpose, information-dense summaries that act as plain-text repositories of web content. Inspired by Hegel's dialectical method, our approach, denoted as Chain of Summaries (CoS), iteratively refines an initial summary (thesis) by identifying its limitations through questioning (antithesis), leading to a general-purpose summary (synthesis) that can satisfy current and anticipate future information needs. Experiments on the TriviaQA, TruthfulQA, and SQUAD datasets demonstrate that CoS outperforms zero-shot LLM baselines by up to 66\% and specialized summarization methods such as Chain of Density, BRIO and PEGASUS by up to 27\%. CoS-generated summaries yield higher Q\&A performance compared to the source content, while requiring substantially fewer tokens and being agnostic to the specific downstream LLM. CoS thus resembles an appealing option for website maintainers to make their content more accessible for LLMs, while retaining possibilities for human oversight.