EmoBench-UA: A Benchmark Dataset for Emotion Detection in Ukrainian

TL;DR

This paper introduces EmoBench-UA, the first annotated dataset for emotion detection in Ukrainian texts, enabling future research in this underexplored area of NLP.

Contribution

It provides a new benchmark dataset for Ukrainian emotion detection and evaluates various approaches, including LLMs and linguistic baselines.

Findings

Emotion classification in Ukrainian is challenging.

English-trained models perform poorly on Ukrainian data.

Highlighting the need for Ukrainian-specific NLP resources.

Abstract

While Ukrainian NLP has seen progress in many texts processing tasks, emotion classification remains an underexplored area with no publicly available benchmark to date. In this work, we introduce EmoBench-UA, the first annotated dataset for emotion detection in Ukrainian texts. Our annotation schema is adapted from the previous English-centric works on emotion detection (Mohammad et al., 2018; Mohammad, 2022) guidelines. The dataset was created through crowdsourcing using the Toloka.ai platform ensuring high-quality of the annotation process. Then, we evaluate a range of approaches on the collected dataset, starting from linguistic-based baselines, synthetic data translated from English, to large language models (LLMs). Our findings highlight the challenges of emotion classification in non-mainstream languages like Ukrainian and emphasize the need for further development of Ukrainian-specific models and training resources.