Evaluating LLMs on Chinese Idiom Translation

TL;DR

This paper introduces IdiomEval, a comprehensive framework for evaluating Chinese idiom translation in large language models, revealing significant translation errors and inadequacies in current metrics, and proposing improved error detection models.

Contribution

The paper presents a new error taxonomy, a large annotated dataset, and improved models for detecting Chinese idiom translation errors in LLMs.

Findings

GPT-4 makes errors in 28% of idiom translations.

Existing metrics correlate poorly with human judgments (Pearson < 0.48).

Developed models achieve F1 score of 0.68 in error detection.

Abstract

Idioms, whose figurative meanings usually differ from their literal interpretations, are common in everyday language, especially in Chinese, where they often contain historical references and follow specific structural patterns. Despite recent progress in machine translation with large language models, little is known about Chinese idiom translation. In this work, we introduce IdiomEval, a framework with a comprehensive error taxonomy for Chinese idiom translation. We annotate 900 translation pairs from nine modern systems, including GPT-4o and Google Translate, across four domains: web, news, Wikipedia, and social media. We find these systems fail at idiom translation, producing incorrect, literal, partial, or even missing translations. The best-performing system, GPT-4, makes errors in 28% of cases. We also find that existing evaluation metrics measure idiom quality poorly with Pearson correlation below 0.48 with human ratings. We thus develop improved models that achieve F$_1$ scores of 0.68 for detecting idiom translation errors.