Beyond Hate Speech: NLP's Challenges and Opportunities in Uncovering Dehumanizing Language

TL;DR

This paper evaluates large language models for detecting dehumanizing language, revealing strengths and disparities, and emphasizing the need for systematic evaluation across different target groups.

Contribution

It provides a comprehensive assessment of LLMs' capabilities in dehumanization detection and highlights existing biases and challenges in the task.

Findings

Only one model (Claude) achieves over 80% F1 in detection.

Performance drops when distinguishing dehumanization from related hate types.

Models show systematic biases across different target groups.

Abstract

Dehumanization, i.e., denying human qualities to individuals or groups, is a particularly harmful form of hate speech that can normalize violence against marginalized communities. Despite advances in NLP for detecting general hate speech, approaches to identifying dehumanizing language remain limited due to scarce annotated data and the subtle nature of such expressions. In this work, we systematically evaluate four state-of-the-art large language models (LLMs) - Claude, GPT, Mistral, and Qwen - for dehumanization detection. Our results show that only one model-Claude-achieves strong performance (over 80% F1) under an optimized configuration, while others, despite their capabilities, perform only moderately. Performance drops further when distinguishing dehumanization from related hate types such as derogation. We also identify systematic disparities across target groups: models tend to over-predict dehumanization for some identities (e.g., Gay men), while under-identifying it for others (e.g., Refugees). These findings motivate the need for systematic, group-level evaluation when applying pretrained language models to dehumanization detection tasks.