Dealing with Annotator Disagreement in Hate Speech Classification

TL;DR

This paper investigates strategies to handle annotator disagreement in hate speech classification, emphasizing the importance of addressing subjectivity in labeling and providing benchmark results for Turkish tweets.

Contribution

It evaluates automatic aggregation methods for multiple annotations in hate speech detection, offering new insights and state-of-the-art benchmarks for Turkish social media data.

Findings

Automatic aggregation improves label quality in hate speech datasets

Addressing annotator disagreement enhances model performance

Provides benchmark results for Turkish hate speech classification

Abstract

Hate speech detection is a crucial task, especially on social media, where harmful content can spread quickly. Implementing machine learning models to automatically identify and address hate speech is essential for mitigating its impact and preventing its proliferation. The first step in developing an effective hate speech detection model is to acquire a high-quality dataset for training. Labeled data is essential for most natural language processing tasks, but categorizing hate speech is difficult due to the diverse and often subjective nature of hate speech, which can lead to varying interpretations and disagreements among annotators. This paper examines strategies for addressing annotator disagreement, an issue that has been largely overlooked. In particular, we evaluate various automatic approaches for aggregating multiple annotations, in the context of hate speech classification in Turkish tweets. Our work highlights the importance of the problem and provides state-of-the-art benchmark results for the detection and understanding of hate speech in online discourse.