Testing the Limits of Machine Translation from One Book

TL;DR

This study evaluates how large language models translate Kanuri, a low-resource language, focusing on domain-specific tasks and the effectiveness of different language resources, highlighting the superiority of parallel sentences for translation quality.

Contribution

The paper introduces a comprehensive evaluation of LLM translation for Kanuri, emphasizing the importance of parallel sentences over grammar or dictionaries in domain-specific contexts.

Findings

Parallel sentences outperform other data sources in translation quality.

Grammar improves zero-shot translation but is not effective alone.

Linguistic accuracy exceeds fluency in LLM translations.

Abstract

Current state-of-the-art models demonstrate capacity to leverage in-context learning to translate into previously unseen language contexts. Tanzer et al. [2024] utilize language materials (e.g. a grammar) to improve translation quality for Kalamang using large language models (LLMs). We focus on Kanuri, a language that, despite having substantial speaker population, has minimal digital resources. We design two datasets for evaluation: one focused on health and humanitarian terms, and another containing generalized terminology, investigating how domain-specific tasks impact LLM translation quality. By providing different combinations of language resources (grammar, dictionary, and parallel sentences), we measure LLM translation effectiveness, comparing results to native speaker translations and human linguist performance. We evaluate using both automatic metrics and native speaker assessments of fluency and accuracy. Results demonstrate that parallel sentences remain the most effective data source, outperforming other methods in human evaluations and automatic metrics. While incorporating grammar improves over zero-shot translation, it fails as an effective standalone data source. Human evaluations reveal that LLMs achieve accuracy (meaning) more effectively than fluency (grammaticality). These findings suggest LLM translation evaluation benefits from multidimensional assessment beyond simple accuracy metrics, and that grammar alone, without parallel sentences, does not provide sufficient context for effective domain-specific translation.