Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance

TL;DR

This paper demonstrates that safety alignment in large language models can be restored using only a single safety example, achieving effective correction without utility loss and with minimal computational cost.

Contribution

The authors introduce a method to fully recover safety alignment in LLMs with just one safety sample, revealing the low-rank safety gradient structure.

Findings

Safety can be restored with one example regardless of model size.

Convergence occurs within a few training epochs.

Method validated across five safety-aligned LLMs and multiple datasets.

Abstract

Fine-tuning safety-aligned large language models (LLMs) can substantially compromise their safety. Previous approaches require many safety samples or calibration sets, which not only incur significant computational overhead during realignment but also lead to noticeable degradation in model utility. Contrary to this belief, we show that safety alignment can be fully recovered with only a single safety example, without sacrificing utility and at minimal cost. Remarkably, this recovery is effective regardless of the number of harmful examples used in fine-tuning or the size of the underlying model, and convergence is achieved within just a few epochs. Furthermore, we uncover the low-rank structure of the safety gradient, which explains why such efficient correction is possible. We validate our findings across five safety-aligned LLMs and multiple datasets, demonstrating the generality of our approach.