State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting?

TL;DR

This study evaluates the performance of large language models versus fine-tuned BERT-like models on text classification tasks across South Slavic languages, highlighting strengths and limitations of LLMs in low-resource language settings.

Contribution

It provides a comprehensive comparison of LLMs and BERT-like models for South Slavic language classification tasks, emphasizing practical considerations and performance insights.

Findings

LLMs often match or surpass fine-tuned models in zero-shot settings.

LLMs perform similarly in South Slavic languages and English.

Fine-tuned BERT-like models are more practical due to lower computational costs.

Abstract

Until recently, fine-tuned BERT-like models provided state-of-the-art performance on text classification tasks. With the rise of instruction-tuned decoder-only models, commonly known as large language models (LLMs), the field has increasingly moved toward zero-shot and few-shot prompting. However, the performance of LLMs on text classification, particularly on less-resourced languages, remains under-explored. In this paper, we evaluate the performance of current language models on text classification tasks across several South Slavic languages. We compare openly available fine-tuned BERT-like models with a selection of open-source and closed-source LLMs across three tasks in three domains: sentiment classification in parliamentary speeches, topic classification in news articles and parliamentary speeches, and genre identification in web texts. Our results show that LLMs demonstrate strong zero-shot performance, often matching or surpassing fine-tuned BERT-like models. Moreover, when used in a zero-shot setup, LLMs perform comparably in South Slavic languages and English. However, we also point out key drawbacks of LLMs, including less predictable outputs, significantly slower inference, and higher computational costs. Due to these limitations, fine-tuned BERT-like models remain a more practical choice for large-scale automatic text annotation.