Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark

TL;DR

This paper introduces DemosQA, a Greek QA dataset, and evaluates 11 monolingual and multilingual LLMs on Greek QA tasks, highlighting the importance of language-specific models for capturing cultural nuances.

Contribution

It presents a new Greek QA dataset, a flexible evaluation framework, and a comprehensive comparison of monolingual and multilingual LLMs for Greek question answering.

Findings

Monolingual Greek LLMs outperform multilingual models on Greek QA tasks.

Prompting strategies significantly influence model performance.

The dataset captures Greek social and cultural context effectively.

Abstract

Recent advancements in Natural Language Processing and Deep Learning have enabled the development of Large Language Models (LLMs), which have significantly advanced the state-of-the-art across a wide range of tasks, including Question Answering (QA). Despite these advancements, research on LLMs has primarily targeted high-resourced languages (e.g., English), and only recently has attention shifted toward multilingual models. However, these models demonstrate a training data bias towards a small number of popular languages or rely on transfer learning from high- to under-resourced languages; this may lead to a misrepresentation of social, cultural, and historical aspects. To address this challenge, monolingual LLMs have been developed for under-resourced languages; however, their effectiveness remains less studied when compared to multilingual counterparts on language-specific tasks. In this study, we address this research gap in Greek QA by contributing: (i) DemosQA, a novel dataset, which is constructed using social media user questions and community-reviewed answers to better capture the Greek social and cultural zeitgeist; (ii) a memory-efficient LLM evaluation framework adaptable to diverse QA datasets and languages; and (iii) an extensive evaluation of 11 monolingual and multilingual LLMs on 6 human-curated Greek QA datasets using 3 different prompting strategies. We release our code and data to facilitate reproducibility.