On the Robustness of Knowledge Editing for Detoxification

TL;DR

This paper critically evaluates the robustness of knowledge editing methods for detoxifying large language models, revealing limitations in their reliability across different scenarios and languages.

Contribution

It introduces a robustness-focused evaluation framework for KE-based detoxification, highlighting common failure modes and limitations in current approaches.

Findings

Pseudo-detoxification is a common failure mode.

Detoxification effectiveness decreases with multiple unsafe edits.

Effectiveness varies across models, objectives, and languages.

Abstract

Knowledge-Editing-based (KE-based) detoxification has emerged as a promising approach for mitigating harmful behaviours in Large Language Models. Existing evaluations, however, largely rely on automatic toxicity classifiers, implicitly assuming that reduced toxicity scores reflect genuine behavioural suppression. In this work, we propose a robustness-oriented evaluation framework for KE-based detoxification that examines its reliability beyond standard classifier-based metrics along three dimensions: optimisation robustness, compositional robustness, and cross-lingual robustness. We identify pseudo-detoxification as a common failure mode, where apparent toxicity reductions arise from degenerate generation behaviours rather than meaningful suppression of unsafe content. We further show that detoxification effectiveness degrades when multiple unsafe behaviours are edited jointly, and that both monolingual and cross-lingual detoxification remain effective only under specific model-method combinations. Overall, our results indicate that KE-based detoxification is robust only for certain models, limited numbers of detoxification objectives, and a subset of languages.