Assessing Language Models' Worldview for Fiction Generation

TL;DR

This paper evaluates how well large language models maintain a consistent worldview necessary for fiction generation, revealing significant limitations and uniform narrative patterns across models.

Contribution

It introduces a method to assess LLMs' worldview consistency and highlights the need for models to better understand and generate coherent fictional story worlds.

Findings

Only two models exhibit consistent worldview

Most models are self-conflicting in their responses

Generated stories show a uniform narrative pattern

Abstract

The use of Large Language Models (LLMs) has become ubiquitous, with abundant applications in computational creativity. One such application is fictional story generation. Fiction is a narrative that occurs in a story world that is slightly different than ours. With LLMs becoming writing partners, we question how suitable they are to generate fiction. This study investigates the ability of LLMs to maintain a state of world essential to generate fiction. Through a series of questions to nine LLMs, we find that only two models exhibit consistent worldview, while the rest are self-conflicting. Subsequent analysis of stories generated by four models revealed a strikingly uniform narrative pattern. This uniformity across models further suggests a lack of `state' necessary for fiction. We highlight the limitations of current LLMs in fiction writing and advocate for future research to test and create story worlds for LLMs to reside in. All code, dataset, and the generated responses can be found in https://github.com/tanny411/llm-reliability-and-consistency-evaluation.