Large-Scale Hate Speech Detection with Cross-Domain Transfer

TL;DR

This paper introduces large-scale, multi-domain hate speech datasets in English and Turkish, demonstrating that Transformer models outperform traditional methods and generalize well across hate domains, with notable transfer learning capabilities.

Contribution

It creates extensive multilingual, multi-domain hate speech datasets and evaluates the effectiveness of Transformer models and cross-domain transfer learning in large-scale hate speech detection.

Findings

Transformer models outperform traditional models by at least 5% in English and 10% in Turkish.

High scalability with 98% and 97% performance recovery using only 20% of training data.

Cross-domain transfer recovers over 90% of performance, with gender and religion domains generalizing better.

Abstract

The performance of hate speech detection models relies on the datasets on which the models are trained. Existing datasets are mostly prepared with a limited number of instances or hate domains that define hate topics. This hinders large-scale analysis and transfer learning with respect to hate domains. In this study, we construct large-scale tweet datasets for hate speech detection in English and a low-resource language, Turkish, consisting of human-labeled 100k tweets per each. Our datasets are designed to have equal number of tweets distributed over five domains. The experimental results supported by statistical tests show that Transformer-based language models outperform conventional bag-of-words and neural models by at least 5% in English and 10% in Turkish for large-scale hate speech detection. The performance is also scalable to different training sizes, such that 98% of performance in English, and 97% in Turkish, are recovered when 20% of training instances are used. We further examine the generalization ability of cross-domain transfer among hate domains. We show that 96% of the performance of a target domain in average is recovered by other domains for English, and 92% for Turkish. Gender and religion are more successful to generalize to other domains, while sports fail most.