A Multi-Task Benchmark for Abusive Language Detection in Low-Resource Settings

TL;DR

This paper introduces a large, annotated multi-task benchmark dataset for abusive language detection in Tigrinya social media, addressing resource scarcity and language diversity challenges in content moderation.

Contribution

It provides the first comprehensive dataset with joint annotations for abusiveness, sentiment, and topic classification in Tigrinya, including Romanized and native scripts, and establishes strong baseline models.

Findings

Small fine-tuned models outperform large language models in low-resource settings.

Achieved 86.67% F1 in abusiveness detection, surpassing LLMs by 7+ points.

Dataset covers 13,717 comments from 7,373 videos across 51 channels.

Abstract

Content moderation research has recently made significant advances, but remains limited in serving the majority of the world's languages due to the lack of resources, leaving millions of vulnerable users to online hostility. This work presents a large-scale human-annotated multi-task benchmark dataset for abusive language detection in Tigrinya social media with joint annotations for three tasks: abusiveness, sentiment, and topic classification. The dataset comprises 13,717 YouTube comments annotated by nine native speakers, collected from 7,373 videos with a total of over 1.2 billion views across 51 channels. We developed an iterative term clustering approach for effective data selection. Recognizing that around 64% of Tigrinya social media content uses Romanized transliterations rather than native Ge'ez script, our dataset accommodates both writing systems to reflect actual language use. We establish strong baselines across the tasks in the benchmark, while leaving significant challenges for future contributions. Our experiments demonstrate that small fine-tuned models outperform prompted frontier large language models (LLMs) in the low-resource setting, achieving 86.67% F1 in abusiveness detection (7+ points over best LLM), and maintain stronger performance in all other tasks. The benchmark is made public to promote research on online safety.