SGHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Singapore

TL;DR

SGHateCheck is a new framework that tests hate speech detection models in Singapore's diverse languages, revealing their weaknesses and guiding improvements for culturally sensitive content moderation.

Contribution

It introduces a culturally adapted testing framework using translation and native annotation, addressing the lack of tools for low-resource languages in hate speech detection.

Findings

Identifies critical flaws in current hate speech models

Highlights inadequacy in sensitive content moderation

Provides a benchmark for low-resource language detection

Abstract

To address the limitations of current hate speech detection models, we introduce \textsf{SGHateCheck}, a novel framework designed for the linguistic and cultural context of Singapore and Southeast Asia. It extends the functional testing approach of HateCheck and MHC, employing large language models for translation and paraphrasing into Singapore's main languages, and refining these with native annotators. \textsf{SGHateCheck} reveals critical flaws in state-of-the-art models, highlighting their inadequacy in sensitive content moderation. This work aims to foster the development of more effective hate speech detection tools for diverse linguistic environments, particularly for Singapore and Southeast Asia contexts.