From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasets

TL;DR

This paper investigates cultural biases in hate speech datasets across different languages and geographies, revealing overrepresentation of certain countries and providing recommendations for more culturally balanced data collection.

Contribution

It introduces a systematic evaluation of cultural bias in hate speech datasets using language and geographical metadata, highlighting biases and suggesting improvements.

Findings

English-language bias in datasets has decreased over recent years.

Hate speech datasets for English, Arabic, and Spanish are overrepresented by specific countries.

Significant geo-cultural bias exists, with datasets overrepresenting US and UK for English.

Abstract

Perceptions of hate can vary greatly across cultural contexts. Hate speech (HS) datasets, however, have traditionally been developed by language. This hides potential cultural biases, as one language may be spoken in different countries home to different cultures. In this work, we evaluate cultural bias in HS datasets by leveraging two interrelated cultural proxies: language and geography. We conduct a systematic survey of HS datasets in eight languages and confirm past findings on their English-language bias, but also show that this bias has been steadily decreasing in the past few years. For three geographically-widespread languages -- English, Arabic and Spanish -- we then leverage geographical metadata from tweets to approximate geo-cultural contexts by pairing language and country information. We find that HS datasets for these languages exhibit a strong geo-cultural bias, largely overrepresenting a handful of countries (e.g., US and UK for English) relative to their prominence in both the broader social media population and the general population speaking these languages. Based on these findings, we formulate recommendations for the creation of future HS datasets.