Generative AI & Fictionality: How Novels Power Large Language Models

TL;DR

This paper investigates how novels and fiction influence large language models like BERT, revealing that fiction's unique attributes shape AI outputs and cultural responses, highlighting the importance of training data sources.

Contribution

It provides an analysis of fiction's role in shaping LLMs and introduces a new perspective on how training data influences AI-generated culture.

Findings

Fictional texts contribute unique attributes to LLMs.

Fiction influences social and communicative qualities of AI outputs.

Training data sources significantly impact AI's cultural and social responses.

Abstract

Generative models, like the one in ChatGPT, are powered by their training data. The models are simply next-word predictors, based on patterns learned from vast amounts of pre-existing text. Since the first generation of GPT, it is striking that the most popular datasets have included substantial collections of novels. For the engineers and research scientists who build these models, there is a common belief that the language in fiction is rich enough to cover all manner of social and communicative phenomena, yet the belief has gone mostly unexamined. How does fiction shape the outputs of generative AI? Specifically, what are novels' effects relative to other forms of text, such as newspapers, Reddit, and Wikipedia? Since the 1970s, literature scholars such as Catherine Gallagher and James Phelan have developed robust and insightful accounts of how fiction operates as a form of discourse and language. Through our study of an influential open-source model (BERT), we find that LLMs leverage familiar attributes and affordances of fiction, while also fomenting new qualities and forms of social response. We argue that if contemporary culture is increasingly shaped by generative AI and machine learning, any analysis of today's various modes of cultural production must account for a relatively novel dimension: computational training data.