MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP

TL;DR

This paper benchmarks 55 large language models on Maltese, a low-resource language, revealing smaller fine-tuned models often outperform LLMs, emphasizing the importance of pre-training exposure and fine-tuning for better NLP performance.

Contribution

Introduces MELABenchv1, a new benchmark for evaluating LLMs on Maltese, and provides insights into factors affecting performance in low-resource language NLP.

Findings

Smaller fine-tuned models outperform many LLMs on Maltese tasks.

Pre-training exposure to Maltese significantly improves model performance.

Fine-tuning offers better performance and lower inference costs despite higher initial investment.

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across various Natural Language Processing (NLP) tasks, largely due to their generalisability and ability to perform tasks without additional training. However, their effectiveness for low-resource languages remains limited. In this study, we evaluate the performance of 55 publicly available LLMs on Maltese, a low-resource language, using a newly introduced benchmark covering 11 discriminative and generative tasks. Our experiments highlight that many models perform poorly, particularly on generative tasks, and that smaller fine-tuned models often perform better across all tasks. From our multidimensional analysis, we investigate various factors impacting performance. We conclude that prior exposure to Maltese during pre-training and instruction-tuning emerges as the most important factor. We also examine the trade-offs between fine-tuning and prompting, highlighting that while fine-tuning requires a higher initial cost, it yields better performance and lower inference costs. Through this work, we aim to highlight the need for more inclusive language technologies and recommend that researchers working with low-resource languages consider more "traditional" language modelling approaches.