HateDebias: On the Diversity and Variability of Hate Speech Debiasing

TL;DR

This paper introduces HateDebias, a benchmark for hate speech debiasing that considers diversity and variability of biases, along with a continual debiasing framework to improve fairness in dynamic real-world environments.

Contribution

It presents a new benchmark dataset capturing diverse and evolving biases in hate speech, and proposes a continual debiasing method to address dynamic bias challenges.

Findings

Our methods improve bias mitigation in dynamic scenarios.

The HateDebias benchmark reveals significant bias-related performance degradation.

Continual debiasing enhances fairness in real-world hate speech detection.

Abstract

Hate speech frequently appears on social media platforms and urgently needs to be effectively controlled. Alleviating the bias caused by hate speech can help resolve various ethical issues. Although existing research has constructed several datasets for hate speech detection, these datasets seldom consider the diversity and variability of bias, making them far from real-world scenarios. To fill this gap, we propose a benchmark HateDebias to analyze the fairness of models under dynamically evolving environments. Specifically, to meet the diversity of biases, we collect hate speech data with different types of biases from real-world scenarios. To further simulate the variability in the real-world scenarios(i.e., the changing of bias attributes in datasets), we construct a dataset to follow the continuous learning setting and evaluate the detection accuracy of models on the HateDebias, where performance degradation indicates a significant bias toward a specific attribute. To provide a potential direction, we further propose a continual debiasing framework tailored to dynamic bias in real-world scenarios, integrating memory replay and bias information regularization to ensure the fairness of the model. Experiment results on the HateDebias benchmark reveal that our methods achieve improved performance in mitigating dynamic biases in real-world scenarios, highlighting the practicality in real-world applications.