Benchmarking Machine Translation with Cultural Awareness

TL;DR

This paper introduces a new culturally-annotated parallel corpus and evaluation metrics to assess the cultural awareness of machine translation systems, highlighting LLMs' superior performance in translating culture-specific items.

Contribution

It presents a novel CSI-enriched corpus and evaluation metrics for Culturally-Aware Machine Translation, enabling better analysis of how different MT systems handle cultural content.

Findings

LLMs outperform neural MT in translating CSIs.

LLMs better leverage external cultural knowledge.

The corpus facilitates future research on cultural aspects in MT.

Abstract

Translating culture-related content is vital for effective cross-cultural communication. However, many culture-specific items (CSIs) often lack viable translations across languages, making it challenging to collect high-quality, diverse parallel corpora with CSI annotations. This difficulty hinders the analysis of cultural awareness of machine translation (MT) systems, including traditional neural MT and the emerging MT paradigm using large language models (LLM). To address this gap, we introduce a novel parallel corpus, enriched with CSI annotations in 6 language pairs for investigating Culturally-Aware Machine Translation--CAMT. Furthermore, we design two evaluation metrics to assess CSI translations, focusing on their pragmatic translation quality. Our findings show the superior ability of LLMs over neural MTs in leveraging external cultural knowledge for translating CSIs, especially those lacking translations in the target culture.