Beyond Accuracy: An Explainability-Driven Analysis of Harmful Content Detection

TL;DR

This paper analyzes a neural harmful content detection model using explainability methods to uncover limitations and failure modes, emphasizing the importance of interpretability for human-in-the-loop moderation beyond mere accuracy metrics.

Contribution

It demonstrates how post-hoc explainability methods reveal model limitations and failure modes in harmful content detection, advocating for transparency over performance metrics.

Findings

Integrated Gradients provide diffuse contextual attributions.

Shapley Additive Explanations focus on explicit lexical cues.

Explainability exposes failure modes like indirect toxicity and political bias.

Abstract

Although automated harmful content detection systems are frequently used to monitor online platforms, moderators and end users frequently cannot understand the logic underlying their predictions. While recent studies have focused on increasing classification accuracy, little focus has been placed on comprehending why neural models identify content as harmful, especially when it comes to borderline, contextual, and politically sensitive situations. In this work, a neural harmful content detection model trained on the Civil Comments dataset is analyzed explainability-drivenly. Two popular post-hoc explanation methods, Shapley Additive Explanations and Integrated Gradients, are used to analyze the behavior of a RoBERTa-based classifier in both correct predictions and systematic failure cases. Despite strong overall performance, with an area under the curve of 0.93 and an accuracy of 0.94, the analysis reveals limitations that are not observable from aggregate evaluation metrics alone. Integrated Gradients appear to extract more diffuse contextual attributions while Shapley Additive Explanations extract more focused attributions on explicit lexical cues. The consequent divergence in their outputs manifests in both false negatives and false positives. Qualitative case studies reveal recurring failure modes such as indirect toxicity, lexical over-attribution, or political discourse. The results suggest that explainable AI can foster human-in-the-loop moderation by exposing model uncertainty and increasing the interpretable rationale behind automated decisions. Most importantly, this work highlights the role of explainability as a transparency and diagnostic resource for online harmful content detection systems rather than as a performance-enhancing lever.