Evaluating Large Language Models on Urdu Idiom Translation

TL;DR

This paper introduces the first evaluation datasets for Urdu idiomatic translation, assesses various LLMs and NMT systems, and finds that prompt engineering and script choice significantly influence translation quality.

Contribution

It provides new datasets for Urdu idiomatic translation and evaluates the impact of prompt engineering and script on translation performance.

Findings

Prompt engineering improves translation quality.

Native Urdu script yields better translations than Roman Urdu.

Translation performance varies with text representation.

Abstract

Idiomatic translation remains a significant challenge in machine translation, especially for low resource languages such as Urdu, and has received limited prior attention. To advance research in this area, we introduce the first evaluation datasets for Urdu to English idiomatic translation, covering both Native Urdu and Roman Urdu scripts and annotated with gold-standard English equivalents. We evaluate multiple open-source Large Language Models (LLMs) and Neural Machine Translation (NMT) systems on this task, focusing on their ability to preserve idiomatic and cultural meaning. Automatic metrics including BLEU, BERTScore, COMET, and XCOMET are used to assess translation quality. Our findings indicate that prompt engineering enhances idiomatic translation compared to direct translation, though performance differences among prompt types are relatively minor. Moreover, cross script comparisons reveal that text representation substantially affects translation quality, with Native Urdu inputs producing more accurate idiomatic translations than Roman Urdu.