NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning

TL;DR

NLSR is a training-free method that restores safety in large language models after fine-tuning by transplanting neurons identified through a reference model, effectively mitigating harmful modifications without retraining.

Contribution

Proposes NLSR, a novel neuron-level safety realignment framework that does not require additional training, using neuron similarity differences to identify and restore safety-critical neurons.

Findings

Significant safety improvements in fine-tuned models across multiple tasks.

Maintains high task accuracy while enhancing safety.

Effective neuron transplantation without additional training.

Abstract

The emergence of finetuning-as-a-service has revealed a new vulnerability in large language models (LLMs). A mere handful of malicious data uploaded by users can subtly manipulate the finetuning process, resulting in an alignment-broken model. Existing methods to counteract fine-tuning attacks typically require substantial computational resources. Even with parameter-efficient techniques like LoRA, gradient updates remain essential. To address these challenges, we propose \textbf{N}euron-\textbf{L}evel \textbf{S}afety \textbf{R}ealignment (\textbf{NLSR}), a training-free framework that restores the safety of LLMs based on the similarity difference of safety-critical neurons before and after fine-tuning. The core of our framework is first to construct a safety reference model from an initially aligned model to amplify safety-related features in neurons. We then utilize this reference model to identify safety-critical neurons, which we prepare as patches. Finally, we selectively restore only those neurons that exhibit significant similarity differences by transplanting these prepared patches, thereby minimally altering the fine-tuned model. Extensive experiments demonstrate significant safety enhancements in fine-tuned models across multiple downstream tasks, while greatly maintaining task-level accuracy. Our findings suggest regions of some safety-critical neurons show noticeable differences after fine-tuning, which can be effectively corrected by transplanting neurons from the reference model without requiring additional training. The code will be available at \url{https://github.com/xinykou/NLSR}