Summarization is (Almost) Dead

TL;DR

Large language models demonstrate superior zero-shot summarization capabilities, often outperforming human and fine-tuned model summaries in factual accuracy and coherence, challenging the necessity of traditional summarization research.

Contribution

The paper introduces new datasets and human evaluation methods to assess LLMs' zero-shot summarization, revealing their dominance over traditional approaches.

Findings

LLMs outperform humans and fine-tuned models in factual consistency

LLMs generate summaries with fewer hallucinations

Traditional summarization research may be less necessary now

Abstract

How well can large language models (LLMs) generate summaries? We develop new datasets and conduct human evaluation experiments to evaluate the zero-shot generation capability of LLMs across five distinct summarization tasks. Our findings indicate a clear preference among human evaluators for LLM-generated summaries over human-written summaries and summaries generated by fine-tuned models. Specifically, LLM-generated summaries exhibit better factual consistency and fewer instances of extrinsic hallucinations. Due to the satisfactory performance of LLMs in summarization tasks (even surpassing the benchmark of reference summaries), we believe that most conventional works in the field of text summarization are no longer necessary in the era of LLMs. However, we recognize that there are still some directions worth exploring, such as the creation of novel datasets with higher quality and more reliable evaluation methods.