Can Good Writing Be Generative? Expert-Level AI Writing Emerges through Fine-Tuning on High-Quality Books

TL;DR

This study shows that fine-tuning AI on high-quality books enables it to emulate expert writing styles convincingly, challenging assumptions about AI's creative limitations and impacting perceptions of good writing.

Contribution

It demonstrates that expert-level AI writing can be achieved through fine-tuning on complete works, significantly narrowing the gap with human authors.

Findings

AI after fine-tuning is preferred over human writing by lay judges.

Experts' preference for AI decreases after fine-tuning, but they still favor human writing.

AI can emulate author styles convincingly, affecting perceptions of creative labor.

Abstract

Creative writing has long been considered a uniquely human endeavor, requiring voice and style that machines could not replicate. This assumption is challenged by Generative AI that can emulate thousands of author styles in seconds with negligible marginal labor. To understand this better, we conducted a behavioral experiment where 28 MFA writers (experts) competed against three LLMs in emulating 50 critically acclaimed authors. Based on blind pairwise comparisons by 28 expert judges and 131 lay judges, we find that experts preferred human writing in 82.7% of cases under the in-context prompting condition but this reversed to 62% preference for AI after fine-tuning on authors' complete works. Lay judges, however, consistently preferred AI writing. Debrief interviews with expert writers revealed that their preference for AI writing triggered an identity crisis, eroding aesthetic confidence and questioning what constitutes "good writing." These findings challenge discourse about AI's creative limitations and raise fundamental questions about the future of creative labor.