Toxic Subword Pruning for Dialogue Response Generation on Large Language Models

TL;DR

This paper introduces ToxPrune, a simple subword pruning method that effectively reduces toxicity in large language models during dialogue generation, while also enhancing response diversity.

Contribution

The paper proposes a novel subword pruning algorithm, ToxPrune, which mitigates toxic content in LLMs without extensive retraining or safety alignment procedures.

Findings

ToxPrune reduces toxic content in LLMs during dialogue generation.

ToxPrune improves dialogue diversity in Llama-3.1-6B.

Automatic and human evaluations confirm ToxPrune's effectiveness.

Abstract

How to defend large language models (LLMs) from generating toxic content is an important research area. Yet, most research focused on various model training techniques to remediate LLMs by updating their weights. A typical related research area is safety alignment. This however is often costly and tedious and can expose the model to even more problems such as catastrophic forgetting if the trainings are not carefully handled by experienced NLP practitioners. We thus propose a simple yet effective and novel algorithm, namely \textbf{Tox}ic Subword \textbf{Prun}ing (ToxPrune) to prune the subword contained by the toxic words from BPE in trained LLMs. In contrast to the previous work that demonstrates pruning BPE tokens as harmful to the task of machine translation, we surprisingly found its usefulness in preventing toxic content from being generated on LLMs. Fortunately, our findings suggest that ToxPrune simultaneously improves the toxic language model NSFW-3B on the task of dialogue response generation obviously. We surprisingly found that ToxPrune can even obviously improve official Llama-3.1-6B in the metric of dialogue diversity. Extensive automatic results and human evaluation indicate that ToxPrune could be helpful for both remediating toxic LLMs and improving non-toxic LLMs on the task of dialogue response generation.\footnote{We plan to release the resources to facilitate future work.}