Echoes in AI: Quantifying lack of plot diversity in LLM outputs

TL;DR

This paper investigates the lack of plot diversity in LLM-generated stories, introduces the Sui Generis score to quantify plot uniqueness, and finds that LLMs tend to echo similar plot elements, reducing creative diversity.

Contribution

The paper introduces the Sui Generis score as a novel automatic metric to measure plot element uniqueness in LLM outputs, highlighting issues of echoing and lack of diversity.

Findings

LLMs often echo plot elements across generations.

Sui Generis score correlates with human surprise judgments.

Human stories rarely have echoed plot elements.

Abstract

With rapid advances in large language models (LLMs), there has been an increasing application of LLMs in creative content ideation and generation. A critical question emerges: can current LLMs provide ideas that are diverse enough to truly bolster collective creativity? We examine two state-of-the-art LLMs, GPT-4 and LLaMA-3, on story generation and discover that LLM-generated stories often consist of plot elements that are echoed across a number of generations. To quantify this phenomenon, we introduce the Sui Generis score, an automatic metric that measures the uniqueness of a plot element among alternative storylines generated using the same prompt under an LLM. Evaluating on 100 short stories, we find that LLM-generated stories often contain combinations of idiosyncratic plot elements echoed frequently across generations and across different LLMs, while plots from the original human-written stories are rarely recreated or even echoed in pieces. Moreover, our human evaluation shows that the ranking of Sui Generis scores among story segments correlates moderately with human judgment of surprise level, even though score computation is completely automatic without relying on human judgment.