PSK@EEUCA 2026: Fine-Tuning Large Language Models with Synthetic Data Augmentation for Multi-Class Toxicity Detection in Gaming Chat

TL;DR

This paper presents a system for toxicity detection in gaming chat using fine-tuned large language models with synthetic data augmentation, achieving competitive results in a shared task.

Contribution

It explores multiple modeling approaches and demonstrates the effectiveness of synthetic data augmentation for multi-class toxicity classification.

Findings

Synthetic data augmentation improves model performance.

Hierarchical and ensemble methods enhance classification accuracy.

Analysis reveals a validation trap affecting generalization.

Abstract

This paper describes our system for the EEUCA 2026 Shared Task on Understanding Toxic Behavior in Gaming Communities. The task involves classifying World of Tanks chat messages into six toxicity categories: Non-toxic, Insults/Flaming, Other Offensive, Hate/Harassment, Threats, and Extremism. We explore multiple approaches including encoder-based models, instruction-tuned LLMs with LoRA fine-tuning, hierarchical classification, one-vs-rest strategies, and various ensemble methods. Our best system combines Llama 3.1 8B with carefully calibrated 5\% synthetic data augmentation, achieving an F1-macro score of 0.6234 on the test set, placing 4th out of 35 participating teams. We provide extensive analysis of the dataset's annotation patterns and their impact on model generalization, revealing a critical ''validation trap'' phenomenon where high validation performance correlates with poor test transfer.