Large Language Models and Book Summarization: Reading or Remembering, Which Is Better?

TL;DR

This paper evaluates whether large language models generate better book summaries from internal knowledge or full text, revealing that sometimes internal knowledge suffices and can outperform full-text summarization.

Contribution

The study provides an experimental comparison of internal knowledge versus full text for book summarization using state-of-the-art LLMs, highlighting their relative strengths.

Findings

Full text generally yields more detailed summaries.

Internal knowledge can outperform full text for some books.

Models' internal knowledge influences summarization quality.

Abstract

Summarization is a core task in Natural Language Processing (NLP). Recent advances in Large Language Models (LLMs) and the introduction of large context windows reaching millions of tokens make it possible to process entire books in a single prompt. At the same time, for well-known books, LLMs can generate summaries based only on internal knowledge acquired during training. This raises several important questions: How do summaries generated from internal memory compare to those derived from the full text? Does prior knowledge influence summaries even when the model is given the book as input? In this work, we conduct an experimental evaluation of book summarization with state-of-the-art LLMs. We compare summaries of well-known books produced using (i) only the internal knowledge of the model and (ii) the full text of the book. The results show that having the full text provides more detailed summaries in general, but some books have better scores for the internal knowledge summaries. This puts into question the capabilities of models to perform summarization of long texts, as information learned during training can outperform summarization of the full text in some cases.