Large Language Models for Persian $ \leftrightarrow $ English Idiom Translation

TL;DR

This study evaluates the effectiveness of various large language models and prompting strategies for translating idiomatic expressions between Persian and English, introducing new datasets and assessing automatic evaluation methods.

Contribution

It presents two parallel idiom datasets for Persian-English translation and systematically compares LLMs, NMT, and their combinations, highlighting the impact of prompting methods.

Findings

Claude-3.5-Sonnet achieves top translation accuracy.

Combining weaker LLMs with Google Translate improves English to Persian translation.

Complex prompts enhance Persian to English translation for advanced models.

Abstract

Large language models (LLMs) have shown superior capabilities in translating figurative language compared to neural machine translation (NMT) systems. However, the impact of different prompting methods and LLM-NMT combinations on idiom translation has yet to be thoroughly investigated. This paper introduces two parallel datasets of sentences containing idiomatic expressions for Persian$\rightarrow$English and English$\rightarrow$Persian translations, with Persian idioms sampled from our PersianIdioms resource, a collection of 2,200 idioms and their meanings, with 700 including usage examples. Using these datasets, we evaluate various open- and closed-source LLMs, NMT models, and their combinations. Translation quality is assessed through idiom translation accuracy and fluency. We also find that automatic evaluation methods like LLM-as-a-judge, BLEU, and BERTScore are effective for comparing different aspects of model performance. Our experiments reveal that Claude-3.5-Sonnet delivers outstanding results in both translation directions. For English$\rightarrow$Persian, combining weaker LLMs with Google Translate improves results, while Persian$\rightarrow$English translations benefit from single prompts for simpler models and complex prompts for advanced ones.