GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning

TL;DR

GR-SAP introduces a generative replay framework that synthesizes alignment data to preserve safety alignment in LLMs during fine-tuning, effectively mitigating safety degradation without sacrificing task performance.

Contribution

The paper proposes GR-SAP, a novel generative replay method that synthesizes alignment data to maintain safety alignment during LLM fine-tuning, addressing data accessibility issues.

Findings

Significantly reduces safety degradation during fine-tuning.

Maintains comparable downstream task performance.

Proven effective across various models and tasks.

Abstract

Recent studies show that the safety alignment of large language models (LLMs) can be easily compromised even by seemingly non-adversarial fine-tuning. To preserve safety alignment during fine-tuning, a widely used strategy is to jointly optimize safety and task objectives by mixing in the original alignment data, which is typically inaccessible even for open-weight LLMs. Inspired by generative replay in continual learning, we propose Generative Replay for Safety Alignment Preservation (GR-SAP), a unified framework that synthesizes domain-specific alignment data from LLMs and integrate them during downstream adaption to preserve safety alignment. Theoretical and empirical analyses demonstrate this synthetic data serves as a reliable proxy for the original alignment data. Experiments across various models and downstream tasks show that GR-SAP substantially mitigates fine-tuning-induced safety degradation while maintaining comparable downstream performance. Our code is available at https://github.com/chili-lab/gr-sap.