Can professional translators identify machine-generated text?

TL;DR

This study examines whether professional Italian translators can reliably distinguish AI-generated stories from human-written ones, revealing limited but notable detection abilities and common misclassification patterns.

Contribution

It provides empirical evidence on the detection capabilities of untrained professional translators regarding AI-generated texts in Italian.

Findings

16.2% of translators successfully identified AI texts

Low burstiness and narrative contradiction are key indicators

Subjective impressions often lead to misclassification

Abstract

This study investigates whether professional translators without prior specialized training can reliably identify short stories generated in Italian by artificial intelligence (AI). Sixty-nine translators took part in an in-person experiment, where they assessed three anonymized short stories - two written by ChatGPT-4o and one by a human author. For each story, participants rated the likelihood of AI authorship and provided justifications for their choices. While average results were inconclusive, a statistically significant subset (16.2%) successfully distinguished the synthetic texts from the human text, suggesting that their judgements were informed by analytical skill rather than chance. However, a nearly equal number misclassified the texts in the opposite direction, often relying on subjective impressions rather than objective markers, possibly reflecting a reader preference for AI-generated texts. Low burstiness and narrative contradiction emerged as the most reliable indicators of synthetic authorship, with unexpected calques, semantic loans and syntactic transfer from English also reported. In contrast, features such as grammatical accuracy and emotional tone frequently led to misclassification. These findings raise questions about the role and scope of synthetic-text editing in professional contexts.