Evaluating LLMs on Chinese Idiom Translation
Cai Yang, Yao Dou, David Heineman, Xiaofeng Wu, Wei Xu

TL;DR
This paper introduces IdiomEval, a comprehensive framework for evaluating Chinese idiom translation in large language models, revealing significant translation errors and inadequacies in current metrics, and proposing improved error detection models.
Contribution
The paper presents a new error taxonomy, a large annotated dataset, and improved models for detecting Chinese idiom translation errors in LLMs.
Findings
GPT-4 makes errors in 28% of idiom translations.
Existing metrics correlate poorly with human judgments (Pearson < 0.48).
Developed models achieve F1 score of 0.68 in error detection.
Abstract
Idioms, whose figurative meanings usually differ from their literal interpretations, are common in everyday language, especially in Chinese, where they often contain historical references and follow specific structural patterns. Despite recent progress in machine translation with large language models, little is known about Chinese idiom translation. In this work, we introduce IdiomEval, a framework with a comprehensive error taxonomy for Chinese idiom translation. We annotate 900 translation pairs from nine modern systems, including GPT-4o and Google Translate, across four domains: web, news, Wikipedia, and social media. We find these systems fail at idiom translation, producing incorrect, literal, partial, or even missing translations. The best-performing system, GPT-4, makes errors in 28% of cases. We also find that existing evaluation metrics measure idiom quality poorly with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
