Exploring Performance Variations in Finetuned Translators of Ultra-Low Resource Languages: Do Linguistic Differences Matter?

TL;DR

This study investigates why finetuned translators for ultra-low resource languages show performance variability, finding linguistic differences between languages may be more influential than training procedures or model size.

Contribution

It systematically examines various training factors and concludes that linguistic differences are likely the key factor affecting translation performance.

Findings

Training procedures and model size have limited impact on performance differences.

Linguistic structural differences between languages significantly influence translation success.

Performance variability is more attributable to language-specific features than to data cleaning or model limitations.

Abstract

Finetuning pre-trained language models with small amounts of data is a commonly-used method to create translators for ultra-low resource languages such as endangered Indigenous languages. However, previous works have reported substantially different performances with translators created using similar methodology and data. In this work we systematically explored possible causes of the performance difference, aiming to determine whether it was a product of different cleaning procedures, limitations of the pre-trained models, the size of the base model, or the size of the training dataset, studying both directions of translation. Our studies, using two Brazilian Indigenous languages, related but with significant structural linguistic characteristics, indicated none or very limited influence from those training factors, suggesting differences between languages may play a significant role in the ability to produce translators by fine-tuning pre-trained models.