Performance of Recent Large Language Models for a Low-Resourced Language

TL;DR

This paper evaluates the performance of recent large language models on the low-resourced Sinhala language, highlighting their strengths and limitations in direct and translated tasks, and assessing the impact of fine-tuning.

Contribution

It provides a comparative analysis of four recent LLMs on Sinhala, including their out-of-the-box performance and potential improvements through fine-tuning.

Findings

Claude and GPT 4o perform well without fine-tuning.

Llama and Mistral perform poorly initially but improve with fine-tuning.

Fine-tuning with small data can enhance Llama and Mistral performance.

Abstract

Large Language Models (LLMs) have shown significant advances in the past year. In addition to new versions of GPT and Llama, several other LLMs have been introduced recently. Some of these are open models available for download and modification. Although multilingual large language models have been available for some time, their performance on low-resourced languages such as Sinhala has been poor. We evaluated four recent LLMs on their performance directly in the Sinhala language, and by translation to and from English. We also evaluated their fine-tunability with a small amount of fine-tuning data. Claude and GPT 4o perform well out-of-the-box and do significantly better than previous versions. Llama and Mistral perform poorly but show some promise of improvement with fine tuning.