Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing

TL;DR

This paper introduces a simple parameter editing method to modulate large language models' behavior, achieving significant detoxification effects without extensive retraining or fine-tuning.

Contribution

The authors demonstrate that editing a small subset of parameters can effectively change LLM behaviors, reducing toxicity and jailbreaking susceptibility with minimal computational cost.

Findings

Achieves up to 90% toxicity reduction on RealToxicityPrompts

Reduces toxicity by 49.2% on ToxiGen dataset

Maintains general LLM capabilities after parameter editing

Abstract

Large Language Models (LLMs) have demonstrated great potential as generalist assistants, showcasing powerful task understanding and problem-solving capabilities. To deploy LLMs as AI assistants, it is crucial that these models exhibit desirable behavioral traits, such as non-toxicity and resilience against jailbreak attempts. Current approaches for detoxification or preventing jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), which requires finetuning billions of parameters through gradient descent with substantial computational cost. Furthermore, models modified through SFT and RLHF may deviate from the pretrained models, potentially leading to a degradation in foundational LLM capabilities. In this paper, we observe that surprisingly, directly editing a small subset of parameters can effectively modulate specific behaviors of LLMs, such as detoxification and resistance to jailbreaking, with only inference-level computational resources. Experiments demonstrate that in the detoxification task, our approach achieves reductions of up to 90.0% in toxicity on the RealToxicityPrompts dataset and 49.2% on ToxiGen, while maintaining the LLM's general capabilities in areas such as common sense, question answering, and mathematics