Do Language Models Agree with Human Perceptions of Suspense in Stories?

TL;DR

This study evaluates how well language models understand and perceive suspense in stories compared to humans, revealing that models can identify suspense presence but fail to match human perception of its intensity and dynamics.

Contribution

It replicates psychological studies on suspense using language models, highlighting their limitations in capturing human-like suspense perception and dynamics.

Findings

LMs can distinguish if a text aims to induce suspense

LMs cannot accurately estimate suspense levels compared to humans

LMs struggle to track suspense fluctuations across story segments

Abstract

Suspense is an affective response to narrative text that is believed to involve complex cognitive processes in humans. Several psychological models have been developed to describe this phenomenon and the circumstances under which text might trigger it. We replicate four seminal psychological studies of human perceptions of suspense, substituting human responses with those of different open-weight and closed-source LMs. We conclude that while LMs can distinguish whether a text is intended to induce suspense in people, LMs cannot accurately estimate the relative amount of suspense within a text sequence as compared to human judgments, nor can LMs properly capture the human perception for the rise and fall of suspense across multiple text segments. We probe the abilities of LM suspense understanding by adversarially permuting the story text to identify what cause human and LM perceptions of suspense to diverge. We conclude that, while LMs can superficially identify and track certain facets of suspense, they do not process suspense in the same way as human readers.