Can Artificial Intelligence Write Like Borges? An Evaluation Protocol for Spanish Microfiction

TL;DR

This paper introduces GrAImes, an evaluation protocol based on literary theory, designed to assess the literary quality of AI-generated microfictions across thematic, aesthetic, and interpretive dimensions.

Contribution

It presents a novel, theory-based evaluation framework for assessing AI-generated microfiction, addressing a gap in literary merit assessment of automated storytelling.

Findings

Validation by literature experts and enthusiasts supports the protocol's effectiveness.

The protocol provides an objective, multi-dimensional assessment of literary qualities.

It offers a foundation for future research in AI-generated literary texts.

Abstract

Automated story writing has been a subject of study for over 60 years. Large language models can generate narratively consistent and linguistically coherent short fiction texts. Despite these advancements, rigorous assessment of such outputs for literary merit - especially concerning aesthetic qualities - has received scant attention. In this paper, we address the challenge of evaluating AI-generated microfictions and argue that this task requires consideration of literary criteria across various aspects of the text, such as thematic coherence, textual clarity, interpretive depth, and aesthetic quality. To facilitate this, we present GrAImes: an evaluation protocol grounded in literary theory, specifically drawing from a literary perspective, to offer an objective framework for assessing AI-generated microfiction. Furthermore, we report the results of our validation of the evaluation protocol, as answered by both literature experts and literary enthusiasts. This protocol will serve as a foundation for evaluating automatically generated microfictions and assessing their literary value.