HateXScore: A Metric Suite for Evaluating Reasoning Quality in Hate Speech Explanations

TL;DR

HateXScore is a comprehensive metric suite designed to evaluate the reasoning quality of hate speech explanations, addressing interpretability and trustworthiness in content moderation.

Contribution

It introduces a novel four-component evaluation framework for reasoning quality in hate speech explanations, filling a gap left by standard metrics.

Findings

HateXScore correlates strongly with human judgments.

It reveals interpretability failures and annotation inconsistencies.

Effective across six diverse hate speech datasets.

Abstract

Hateful speech detection is a key component of content moderation, yet current evaluation frameworks rarely assess why a text is deemed hateful. We introduce \textsf{HateXScore}, a four-component metric suite designed to evaluate the reasoning quality of model explanations. It assesses (i) conclusion explicitness, (ii) faithfulness and causal grounding of quoted spans, (iii) protected group identification (policy-configurable), and (iv) logical consistency among these elements. Evaluated on six diverse hate speech datasets, \textsf{HateXScore} is intended as a diagnostic complement to reveal interpretability failures and annotation inconsistencies that are invisible to standard metrics like Accuracy or F1. Moreover, human evaluation shows strong agreement with \textsf{HateXScore}, validating it as a practical tool for trustworthy and transparent moderation. \textcolor{red}{Disclaimer: This paper contains sensitive content that may be disturbing to some readers.}