Can Peter Pan Survive MT? A Stylometric Study of LLMs, NMTs, and HTs in Children's Literature Translation

TL;DR

This paper evaluates the stylistic differences between human and machine translations of children's literature, specifically Peter Pan, using a comprehensive stylometric analysis to assess the performance of LLMs and NMTs in capturing human-like translation qualities.

Contribution

It introduces a novel stylometric framework with 447 features to compare human, LLM, and NMT translations of children's literature, highlighting the potential of LLMs to produce more human-like translations.

Findings

NMTs and LLMs differ significantly in descriptive word usage.

LLMs align more closely with human translations in stylistic features.

Stylistic differences are evident in lexical, syntactic, and CTT-specific features.

Abstract

This study focuses on evaluating the performance of machine translations (MTs) compared to human translations (HTs) in English-to-Chinese children's literature translation (CLT) from a stylometric perspective. The research constructs a Peter Pan corpus, comprising 21 translations: 7 human translations (HTs), 7 large language model translations (LLMs), and 7 neural machine translation outputs (NMTs). The analysis employs a generic feature set (including lexical, syntactic, readability, and n-gram features) and a creative text translation (CTT-specific) feature set, which captures repetition, rhythm, translatability, and miscellaneous levels, yielding 447 linguistic features in total. Using classification and clustering techniques in machine learning, we conduct a stylometric analysis of these translations. Results reveal that in generic features, HTs and MTs exhibit significant differences in conjunction word distributions and the ratio of 1-word-gram-YiYang, while NMTs and LLMs show significant variation in descriptive words usage and adverb ratios. Regarding CTT-specific features, LLMs outperform NMTs in distribution, aligning more closely with HTs in stylistic characteristics, demonstrating the potential of LLMs in CLT.