Evaluating OpenAI GPT Models for Translation of Endangered Uralic Languages: A Comparison of Reasoning and Non-Reasoning Architectures

TL;DR

This paper compares reasoning and non-reasoning GPT models in translating endangered Uralic languages, revealing that reasoning models have lower refusal rates and potentially better performance for language preservation efforts.

Contribution

It provides a novel comparison of GPT architectures for low-resource language translation, highlighting reasoning models' advantages in refusal rates and translation quality.

Findings

Reasoning models show 16% lower refusal rates.

Significant performance differences between model types.

Insights for endangered language translation and preservation.

Abstract

The evaluation of Large Language Models (LLMs) for translation tasks has primarily focused on high-resource languages, leaving a significant gap in understanding their performance on low-resource and endangered languages. This study presents a comprehensive comparison of OpenAI's GPT models, specifically examining the differences between reasoning and non-reasoning architectures for translating between Finnish and four low-resource Uralic languages: Komi-Zyrian, Moksha, Erzya, and Udmurt. Using a parallel corpus of literary texts, we evaluate model willingness to attempt translation through refusal rate analysis across different model architectures. Our findings reveal significant performance variations between reasoning and non-reasoning models, with reasoning models showing 16 percentage points lower refusal rates. The results provide valuable insights for researchers and practitioners working with Uralic languages and contribute to the broader understanding of reasoning model capabilities for endangered language preservation.