The author is dead, but what if they never lived? A reception experiment on Czech AI- and human-authored poetry

TL;DR

This study investigates Czech AI- and human-authored poetry perception, revealing that readers often cannot distinguish AI from human poems and that beliefs about authorship influence aesthetic judgments, highlighting AI's growing poetic capabilities in low-resource languages.

Contribution

It provides empirical evidence that Czech AI-generated poetry is indistinguishable from human poetry and explores how authorship perception affects aesthetic evaluation in a low-resource language context.

Findings

Participants guessed authorship at chance level (45.8%).

AI poems were rated as favorably as or more favorably than human poems.

Belief that a poem is AI-generated reduces its aesthetic rating.

Abstract

Large language models are increasingly capable of producing creative texts, yet most studies on AI-generated poetry focus on English -- a language that dominates training data. In this paper, we examine the perception of AI- and human-written Czech poetry. We ask if Czech native speakers are able to identify it and how they aesthetically judge it. Participants performed at chance level when guessing authorship (45.8\% correct on average), indicating that Czech AI-generated poems were largely indistinguishable from human-written ones. Aesthetic evaluations revealed a strong authorship bias: when participants believed a poem was AI-generated, they rated it as less favorably, even though AI poems were in fact rated equally or more favorably than human ones on average. The logistic regression model uncovered that the more the people liked a poem, the less probable was that they accurately assign the authorship. Familiarity with poetry or literary background had no effect on recognition accuracy. Our findings show that AI can convincingly produce poetry even in a morphologically complex, low-resource (with respect of the training data of AI models) Slavic language such as Czech. The results suggest that readers' beliefs about authorship and the aesthetic evaluation of the poem are interconnected.