IndoToxic2024: A Demographically-Enriched Dataset of Hate Speech and Toxicity Types for Indonesian Language

TL;DR

This paper introduces IndoToxic2024, a large, annotated dataset of Indonesian hate speech targeting vulnerable groups, and demonstrates baseline classification results and the impact of demographic info on model performance.

Contribution

The creation of IndoToxic2024, a comprehensive, demographically-enriched Indonesian hate speech dataset, and the evaluation of baseline models and demographic information integration.

Findings

Achieved a macro-F1 score of 0.78 with IndoBERTweet on hate speech classification.

Demographic information improves zero-shot performance of gpt-3.5-turbo.

Overemphasis on demographic data can reduce fine-tuned model performance.

Abstract

Hate speech poses a significant threat to social harmony. Over the past two years, Indonesia has seen a ten-fold increase in the online hate speech ratio, underscoring the urgent need for effective detection mechanisms. However, progress is hindered by the limited availability of labeled data for Indonesian texts. The condition is even worse for marginalized minorities, such as Shia, LGBTQ, and other ethnic minorities because hate speech is underreported and less understood by detection tools. Furthermore, the lack of accommodation for subjectivity in current datasets compounds this issue. To address this, we introduce IndoToxic2024, a comprehensive Indonesian hate speech and toxicity classification dataset. Comprising 43,692 entries annotated by 19 diverse individuals, the dataset focuses on texts targeting vulnerable groups in Indonesia, specifically during the hottest political event in the country: the presidential election. We establish baselines for seven binary classification tasks, achieving a macro-F1 score of 0.78 with a BERT model (IndoBERTweet) fine-tuned for hate speech classification. Furthermore, we demonstrate how incorporating demographic information can enhance the zero-shot performance of the large language model, gpt-3.5-turbo. However, we also caution that an overemphasis on demographic information can negatively impact the fine-tuned model performance due to data fragmentation.