Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation

TL;DR

This paper introduces a paired-task framework to evaluate large language models' abilities in both understanding source texts and generating creative translations, revealing significant gaps in AI creativity compared to humans.

Contribution

It presents a novel paired-task evaluation method combining human and automatic scoring to benchmark LLMs on comprehension and creativity in literary translation.

Findings

Strong comprehension does not lead to human-level creativity.

Models often produce literal or inappropriate translations, especially in English-Chinese pairs.

Only one model, Mistral-Large, approaches human creativity scores.

Abstract

Large language models (LLMs) are increasingly used for creative tasks such as literary translation. Yet translational creativity remains underexplored and is rarely evaluated at scale, while source-text comprehension is typically studied in isolation, despite the fact that, in professional translation, comprehension and creativity are tightly intertwined. We address these gaps with a paired-task framework applied to literary excerpts from 11 books. Task 1 assesses source-text comprehension, and Task 2 evaluates translational creativity through Units of Creative Potential (UCPs), such as metaphors and wordplay. Using a scalable evaluation setup that combines expert human annotations with UCP-based automatic scoring, we benchmark 23 models and four creativity-oriented prompts. Our findings show that strong comprehension does not translate into human-level creativity: models often produce literal or contextually inappropriate renderings, with particularly large gaps for the more distant English-Chinese language pair. Creativity-oriented prompts yield only modest gains, and only one model, Mistral-Large, comes close to human-level creativity (0.167 vs. 0.246). Across all model-prompt combinations, only three exceed a creativity score of 0.1, while the rest remain at or near zero.