Large Language Models for Toxic Language Detection in Low-Resource Balkan Languages

TL;DR

This study evaluates large language models for detecting toxic language in low-resource Balkan languages, demonstrating that minimal context and prompt design improvements can significantly enhance detection performance.

Contribution

It introduces a new dataset and systematically compares LLMs in zero-shot and context-augmented modes for toxicity detection in Balkan languages, highlighting practical strategies for improvement.

Findings

Adding context improves recall and F1 score.

Gemini in context mode achieves F1 of 0.82.

GPT-4.1 zero-shot has highest precision.

Abstract

Online toxic language causes real harm, especially in regions with limited moderation tools. In this study, we evaluate how large language models handle toxic comments in Serbian, Croatian, and Bosnian, languages with limited labeled data. We built and manually labeled a dataset of 4,500 YouTube and TikTok comments drawn from videos across diverse categories, including music, politics, sports, modeling, influencer content, discussions of sexism, and general topics. Four models (GPT-3.5 Turbo, GPT-4.1, Gemini 1.5 Pro, and Claude 3 Opus) were tested in two modes: zero-shot and context-augmented. We measured precision, recall, F1 score, accuracy and false positive rates. Including a short context snippet raised recall by about 0.12 on average and improved F1 score by up to 0.10, though it sometimes increased false positives. The best balance came from Gemini in context-augmented mode, reaching an F1 score of 0.82 and accuracy of 0.82, while zero-shot GPT-4.1 led on precision and had the lowest false alarms. We show how adding minimal context can improve toxic language detection in low-resource settings and suggest practical strategies such as improved prompt design and threshold calibration. These results show that prompt design alone can yield meaningful gains in toxicity detection for underserved Balkan language communities.