Mitigating Biases to Embrace Diversity: A Comprehensive Annotation Benchmark for Toxic Language

TL;DR

This paper presents a new annotation benchmark for toxic language that reduces bias, improves annotation consistency, and demonstrates the effectiveness of LLMs as annotation tools, especially when data is limited or diverse language is involved.

Contribution

It introduces two annotated datasets with higher agreement and shows that fine-tuned smaller models on multi-source LLM data outperform larger models trained on single-source human data.

Findings

LLMs can effectively replace human annotators.

Structured guidelines reduce subjective variability.

Smaller models with multi-source data outperform larger single-source models.

Abstract

This study introduces a prescriptive annotation benchmark grounded in humanities research to ensure consistent, unbiased labeling of offensive language, particularly for casual and non-mainstream language uses. We contribute two newly annotated datasets that achieve higher inter-annotator agreement between human and language model (LLM) annotations compared to original datasets based on descriptive instructions. Our experiments show that LLMs can serve as effective alternatives when professional annotators are unavailable. Moreover, smaller models fine-tuned on multi-source LLM-annotated data outperform models trained on larger, single-source human-annotated datasets. These findings highlight the value of structured guidelines in reducing subjective variability, maintaining performance with limited data, and embracing language diversity. Content Warning: This article only analyzes offensive language for academic purposes. Discretion is advised.