Detoxification of Large Language Models through Output-layer Fusion with a Calibration Model

TL;DR

This paper introduces a lightweight calibration-based method to detoxify large language models, effectively reducing harmful outputs without sacrificing fluency or requiring extensive retraining.

Contribution

The authors propose a novel output-layer fusion approach using a pre-trained calibration model for efficient LLM detoxification, avoiding costly retraining or prompt engineering.

Findings

Reduces toxicity effectively on benchmark datasets

Maintains LLM fluency and contextual understanding

Seamlessly applies to multiple LLMs without retraining

Abstract

Existing approaches for Large language model (LLM) detoxification generally rely on training on large-scale non-toxic or human-annotated preference data, designing prompts to instruct the LLM to generate safe content, or modifying the model parameters to remove toxic information, which are computationally expensive, lack robustness, and often compromise LLMs' fluency and contextual understanding. In this paper, we propose a simple yet effective approach for LLM detoxification, which leverages a compact, pre-trained calibration model that guides the detoxification process of a target LLM via a lightweight intervention in its generation pipeline. By learning a detoxified embedding space from non-toxic data, the calibration model effectively steers the LLM away from generating harmful content. This approach only requires a one-time training of the calibration model that is able to be seamlessly applied to multiple LLMs without compromising fluency or contextual understanding. Experiment results on the benchmark dataset demonstrate that our approach reduces toxicity while maintaining reasonable content expression.