From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation

TL;DR

This paper introduces CulT-Eval, a comprehensive benchmark for evaluating machine translation models on culturally grounded expressions, revealing current models' struggles with preserving cultural nuances and proposing improved evaluation metrics.

Contribution

The paper presents CulT-Eval, a new benchmark with a systematic framework and error taxonomy for assessing translation of culture-specific expressions in machine translation.

Findings

Current models often fail to preserve cultural meanings.

Existing metrics overlook culturally induced meaning deviations.

CulT-Eval reveals systematic failure modes in translation models.

Abstract

Culture-expressions, such as idioms, slang, and culture-specific items (CSIs), are pervasive in natural language and encode meanings that go beyond literal linguistic form. Accurately translating such expressions remains challenging for machine translation systems. Despite this, existing benchmarks remain fragmented and do not provide a systematic framework for evaluating translation performance on culture-loaded expressions. To address this gap, we introduce CulT-Eval, a benchmark designed to evaluate how models handle different types of culturally grounded expressions. CulT-Eval comprises over 7,959 carefully curated instances spanning multiple types of culturally grounded expressions, with a comprehensive error taxonomy covering culturally grounded expressions. Through extensive evaluation of large language models and detailed analysis, we identify recurring and systematic failure modes that are not adequately captured by existing automatic metrics. Accordingly, we propose a complementary evaluation metric that targets culturally induced meaning deviations overlooked by standard MT metrics. The results indicate that current models struggle to preserve culturally grounded meaning and to capture the cultural and contextual nuances essential for accurate translation. Our benchmark and code are available at https://anonymous.4open.science/r/CulT-Eval-E75D/.