ModelCitizens: Representing Community Voices in Online Safety

TL;DR

This paper introduces MODELCITIZENS, a community-informed dataset for toxicity detection on social media, highlighting the importance of diverse perspectives and context in improving moderation tools.

Contribution

It presents a new dataset with community-specific annotations, augmented conversational context, and fine-tuned models that outperform existing toxicity detection tools.

Findings

State-of-the-art tools underperform on community-informed data

Context augmentation degrades detection accuracy

Fine-tuned models outperform GPT-o4-mini by 5.5%

Abstract

Automatic toxic language detection is critical for creating safe, inclusive online spaces. However, it is a highly subjective task, with perceptions of toxic language shaped by community norms and lived experience. Existing toxicity detection models are typically trained on annotations that collapse diverse annotator perspectives into a single ground truth, erasing important context-specific notions of toxicity such as reclaimed language. To address this, we introduce MODELCITIZENS, a dataset of 6.8K social media posts and 40K toxicity annotations across diverse identity groups. To capture the role of conversational context on toxicity, typical of social media posts, we augment MODELCITIZENS posts with LLM-generated conversational scenarios. State-of-the-art toxicity detection tools (e.g. OpenAI Moderation API, GPT-o4-mini) underperform on MODELCITIZENS, with further degradation on context-augmented posts. Finally, we release LLAMACITIZEN-8B and GEMMACITIZEN-12B, LLaMA- and Gemma-based models finetuned on MODELCITIZENS, which outperform GPT-o4-mini by 5.5% on in-distribution evaluations. Our findings highlight the importance of community-informed annotation and modeling for inclusive content moderation. The data, models and code are available at https://github.com/asuvarna31/modelcitizens.