Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation

TL;DR

This paper introduces CanMT, a new dataset and evaluation framework for assessing culture-aware translation in large language models, revealing performance disparities and the importance of reference translations.

Contribution

It provides a novel dataset and evaluation framework specifically designed for culture-aware machine translation in LLMs, along with systematic analysis of model behaviors.

Findings

Significant performance differences across models in culture-aware translation.

Translation strategies systematically influence model behavior.

Reference translations improve evaluation reliability.

Abstract

Large language models (LLMs) have achieved strong performance in general machine translation, yet their ability in culture-aware scenarios remains poorly understood. To bridge this gap, we introduce CanMT, a Culture-Aware Novel-Driven Parallel Dataset for Machine Translation, together with a theoretically grounded, multi-dimensional evaluation framework for assessing cultural translation quality. Leveraging CanMT, we systematically evaluate a wide range of LLMs and translation systems under different translation strategy constraints. Our findings reveal substantial performance disparities across models and demonstrate that translation strategies exert a systematic influence on model behavior. Further analysis shows that translation difficulty varies across types of culture-specific items, and that a persistent gap remains between models' recognition of culture-specific knowledge and their ability to correctly operationalize it in translation outputs. In addition, incorporating reference translations is shown to substantially improve evaluation reliability in LLM-as-a-judge, underscoring their essential role in assessing culture-aware translation quality. The corpus and code are available at CanMT.