LiteraryTaste: A Preference Dataset for Creative Writing Personalization

TL;DR

LiteraryTaste introduces a dataset capturing individual reading preferences to enhance personalization in creative writing models, revealing diverse tastes and limited correlation between stated and actual preferences.

Contribution

The paper presents LiteraryTaste, a novel dataset with self-reported and annotated preferences, and demonstrates how fine-tuned transformers can model personal creative writing tastes.

Findings

People have diverse creative writing preferences.

Fine-tuned transformers achieve over 75% accuracy in modeling preferences.

Stated preferences are less predictive of actual preferences.

Abstract

People have different creative writing preferences, and large language models (LLMs) for these tasks can benefit from adapting to each user's preferences. However, these models are often trained over a dataset that considers varying personal tastes as a monolith. To facilitate developing personalized creative writing LLMs, we introduce LiteraryTaste, a dataset of reading preferences from 60 people, where each person: 1) self-reported their reading habits and tastes (stated preference), and 2) annotated their preferences over 100 pairs of short creative writing texts (revealed preference). With our dataset, we found that: 1) people diverge on creative writing preferences, 2) finetuning a transformer encoder could achieve 75.8% and 67.7% accuracy when modeling personal and collective revealed preferences, and 3) stated preferences had limited utility in modeling revealed preferences. With an LLM-driven interpretability pipeline, we analyzed how people's preferences vary. We hope our work serves as a cornerstone for personalizing creative writing technologies.