Proverbs Run in Pairs: Evaluating Proverb Translation Capability of Large Language Model

TL;DR

This paper evaluates the ability of large language models and neural machine translation systems to translate culturally rooted proverbs across languages, revealing strengths in similar cultures and limitations of current evaluation metrics.

Contribution

It introduces a new proverb translation dataset and compares LLMs with NMT models, highlighting the superior performance of LLMs and the inadequacy of existing automatic evaluation metrics.

Findings

LLMs outperform NMT in proverb translation.

Models perform better between culturally similar languages.

Current automatic metrics are unreliable for proverb translation quality.

Abstract

Despite achieving remarkable performance, machine translation (MT) research remains underexplored in terms of translating cultural elements in languages, such as idioms, proverbs, and colloquial expressions. This paper investigates the capability of state-of-the-art neural machine translation (NMT) and large language models (LLMs) in translating proverbs, which are deeply rooted in cultural contexts. We construct a translation dataset of standalone proverbs and proverbs in conversation for four language pairs. Our experiments show that the studied models can achieve good translation between languages with similar cultural backgrounds, and LLMs generally outperform NMT models in proverb translation. Furthermore, we find that current automatic evaluation metrics such as BLEU, CHRF++ and COMET are inadequate for reliably assessing the quality of proverb translation, highlighting the need for more culturally aware evaluation metrics.