Are Large Language Models Capable of Generating Human-Level Narratives?

TL;DR

This study evaluates large language models' storytelling abilities, revealing significant gaps compared to humans in narrative complexity, tension, and emotional depth, and proposes methods to improve their narrative generation.

Contribution

Introduces a novel computational framework for analyzing narrative discourse aspects and demonstrates how explicit discourse feature integration enhances LLM storytelling.

Findings

LLMs produce less suspenseful and more positive stories than humans.

Most LLMs lack discourse understanding and narrative reasoning skills.

Explicit discourse feature integration improves storytelling diversity and emotional engagement by over 40%.

Abstract

This paper investigates the capability of LLMs in storytelling, focusing on narrative development and plot progression. We introduce a novel computational framework to analyze narratives through three discourse-level aspects: i) story arcs, ii) turning points, and iii) affective dimensions, including arousal and valence. By leveraging expert and automatic annotations, we uncover significant discrepancies between the LLM- and human- written stories. While human-written stories are suspenseful, arousing, and diverse in narrative structures, LLM stories are homogeneously positive and lack tension. Next, we measure narrative reasoning skills as a precursor to generative capacities, concluding that most LLMs fall short of human abilities in discourse understanding. Finally, we show that explicit integration of aforementioned discourse features can enhance storytelling, as is demonstrated by over 40% improvement in neural storytelling in terms of diversity, suspense, and arousal.