skLEP: A Slovak General Language Understanding Benchmark

TL;DR

This paper introduces skLEP, a comprehensive Slovak language understanding benchmark with diverse tasks, datasets, and evaluations to advance Slovak NLP research and model development.

Contribution

It presents the first Slovak-specific NLU benchmark, including datasets, evaluation of models, and open-source tools to promote reproducibility and progress.

Findings

Extensive evaluation of Slovak, multilingual, and English models on skLEP tasks.

Creation of original Slovak datasets and translation of English resources.

Public release of benchmark data, tools, and leaderboard.

Abstract

In this work, we introduce skLEP, the first comprehensive benchmark specifically designed for evaluating Slovak natural language understanding (NLU) models. We have compiled skLEP to encompass nine diverse tasks that span token-level, sentence-pair, and document-level challenges, thereby offering a thorough assessment of model capabilities. To create this benchmark, we curated new, original datasets tailored for Slovak and meticulously translated established English NLU resources. Within this paper, we also present the first systematic and extensive evaluation of a wide array of Slovak-specific, multilingual, and English pre-trained language models using the skLEP tasks. Finally, we also release the complete benchmark data, an open-source toolkit facilitating both fine-tuning and evaluation of models, and a public leaderboard at https://github.com/slovak-nlp/sklep in the hopes of fostering reproducibility and drive future research in Slovak NLU.