Concept-Based Interpretability for Toxicity Detection

TL;DR

This paper introduces a concept-based interpretability method for toxicity detection that uses concept gradients and lexicon analysis to improve understanding of model decisions and address over-attribution issues.

Contribution

It proposes a novel interpretability technique using Concept Gradient methods and lexicon-based analysis to better understand toxicity detection models and mitigate over-attribution biases.

Findings

Concept Gradient provides causal insights into toxicity classification.

Lexicon-based analysis identifies words contributing to misclassification.

Lexicon-free augmentation reduces over-attribution of toxic concepts.

Abstract

The rise of social networks has not only facilitated communication but also allowed the spread of harmful content. Although significant advances have been made in detecting toxic language in textual data, the exploration of concept-based explanations in toxicity detection remains limited. In this study, we leverage various subtype attributes present in toxicity detection datasets, such as obscene, threat, insult, identity attack, and sexual explicit as concepts that serve as strong indicators to identify whether language is toxic. However, disproportionate attribution of concepts towards the target class often results in classification errors. Our work introduces an interpretability technique based on the Concept Gradient (CG) method which provides a more causal interpretation by measuring how changes in concepts directly affect the output of the model. This is an extension of traditional gradient-based methods in machine learning, which often focus solely on input features. We propose the curation of Targeted Lexicon Set, which captures toxic words that contribute to misclassifications in text classification models. To assess the significance of these lexicon sets in misclassification, we compute Word-Concept Alignment (WCA) scores, which quantify the extent to which these words lead to errors due to over-attribution to toxic concepts. Finally, we introduce a lexicon-free augmentation strategy by generating toxic samples that exclude predefined toxic lexicon sets. This approach allows us to examine whether over-attribution persists when explicit lexical overlap is removed, providing insights into the model's attribution on broader toxic language patterns.