Text Detoxification: Data Efficiency, Semantic Preservation and Model Generalization

TL;DR

This paper introduces a two-stage training framework for text detoxification that enhances data efficiency, preserves semantics, and improves model robustness against out-of-distribution toxic content, outperforming previous methods.

Contribution

The authors propose a novel two-stage training approach combining supervised fine-tuning with reward-based training to improve detoxification performance and data efficiency.

Findings

Achieves state-of-the-art detoxification results

Reduces reliance on costly annotated data

Enhances model generalization to out-of-distribution inputs

Abstract

The widespread dissemination of toxic content on social media poses a serious threat to both online environments and public discourse, highlighting the urgent need for detoxification methods that effectively remove toxicity while preserving the original semantics. However, existing approaches often struggle to simultaneously achieve strong detoxification performance, semantic preservation, and robustness to out-of-distribution data. Moreover, they typically rely on costly, manually annotated parallel corpora while showing poor data efficiency. To address these challenges, we propose a two-stage training framework that jointly optimizes for data efficiency, semantic preservation, and model generalization. We first perform supervised fine-tuning on a small set of high-quality, filtered parallel data to establish a strong initialization. Then, we leverage unlabeled toxic inputs and a custom-designed reward model to train the LLM using Group Relative Policy Optimization. Experimental results demonstrate that our method effectively mitigates the trade-offs faced by previous work, achieving state-of-the-art performance with improved generalization and significantly reduced dependence on annotated data. Our code is available at: https://github.com/allacnobug/Detoxification-of-Text.