Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries

TL;DR

This paper compares human and LLM-generated novel summaries to evaluate how well models capture long-form narrative understanding, revealing differences in focus and highlighting challenges in long-text comprehension.

Contribution

It introduces a novel alignment-based method to analyze LLMs' narrative engagement and provides a dataset for future research on long-form text summarization.

Findings

Models emphasize the ends of texts more than humans do.

Alignments reveal the complexity of summarization and narrative focus.

Differences in stylistic and focus patterns between humans and models.

Abstract

Although LLM context lengths have grown, there is evidence that their ability to integrate information across long-form texts has not kept pace. We evaluate one such understanding task: generating summaries of novels. When human authors of summaries compress a story, they reveal what they consider narratively important. Therefore, by comparing human and LLM-authored summaries, we can assess whether models mirror human patterns of conceptual engagement with texts. To measure conceptual engagement, we align sentences from 150 human-written novel summaries with the specific chapters they reference. We demonstrate the difficulty of this alignment task, which indicates the complexity of summarization as a task. We then generate and align additional summaries by nine state-of-the-art LLMs for each of the 150 reference texts. Comparing the human and model-authored summaries, we find both stylistic differences between the texts and differences in how humans and LLMs distribute their focus throughout a narrative, with models emphasizing the ends of texts. Comparing human narrative engagement with model attention mechanisms suggests explanations for degraded narrative comprehension and targets for future development. We release our dataset to support future research.