Revisiting Hate Speech Benchmarks: From Data Curation to System Deployment

TL;DR

This paper introduces GOTHate, a large, diverse, code-mixed hate speech dataset from Twitter, and proposes HEN-mBERT, a multilingual model leveraging endogenous signals to improve hate speech detection in real-world scenarios.

Contribution

The paper presents GOTHate, a novel, neutrally-seeded, multilingual hate speech dataset, and introduces HEN-mBERT, a modular model that incorporates endogenous signals for enhanced detection.

Findings

GOTHate is more challenging to classify than existing datasets.

Adding endogenous signals improves hate speech detection accuracy.

HEN-mBERT outperforms baseline models by 2.5% in macro-F1 and 5% in hate class F1.

Abstract

Social media is awash with hateful content, much of which is often veiled with linguistic and topical diversity. The benchmark datasets used for hate speech detection do not account for such divagation as they are predominantly compiled using hate lexicons. However, capturing hate signals becomes challenging in neutrally-seeded malicious content. Thus, designing models and datasets that mimic the real-world variability of hate warrants further investigation. To this end, we present GOTHate, a large-scale code-mixed crowdsourced dataset of around 51k posts for hate speech detection from Twitter. GOTHate is neutrally seeded, encompassing different languages and topics. We conduct detailed comparisons of GOTHate with the existing hate speech datasets, highlighting its novelty. We benchmark it with 10 recent baselines. Our extensive empirical and benchmarking experiments suggest that GOTHate is hard to classify in a text-only setup. Thus, we investigate how adding endogenous signals enhances the hate speech detection task. We augment GOTHate with the user's timeline information and ego network, bringing the overall data source closer to the real-world setup for understanding hateful content. Our proposed solution HEN-mBERT is a modular, multilingual, mixture-of-experts model that enriches the linguistic subspace with latent endogenous signals from history, topology, and exemplars. HEN-mBERT transcends the best baseline by 2.5% and 5% in overall macro-F1 and hate class F1, respectively. Inspired by our experiments, in partnership with Wipro AI, we are developing a semi-automated pipeline to detect hateful content as a part of their mission to tackle online harm.