A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls

TL;DR

This paper introduces a two-stage evaluation framework for literary machine translation from English to Korean, offering more interpretable metrics that better align with human judgments but still face challenges in cultural nuances.

Contribution

The study presents a novel two-step pipeline for literary translation evaluation, improving correlation with human assessments and highlighting limitations of current metrics.

Findings

Framework provides fine-grained, interpretable metrics.

Higher correlation with human judgment than traditional metrics.

Challenges remain in evaluating cultural and honorific aspects.

Abstract

In this work, we propose and evaluate the feasibility of a two-stage pipeline to evaluate literary machine translation, in a fine-grained manner, from English to Korean. The results show that our framework provides fine-grained, interpretable metrics suited for literary translation and obtains a higher correlation with human judgment than traditional machine translation metrics. Nonetheless, it still fails to match inter-human agreement, especially in metrics like Korean Honorifics. We also observe that LLMs tend to favor translations generated by other LLMs, and we highlight the necessity of developing more sophisticated evaluation methods to ensure accurate and culturally sensitive machine translation of literary works.