Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment

TL;DR

This paper introduces LARF, a layer-aware filtering method that detects and removes safety-degrading samples from fine-tuning datasets, thereby preserving the safety alignment of large language models during adaptation.

Contribution

LARF is a novel approach that identifies safety-sensitive layers in LLMs to filter out unsafe training data, improving safety during fine-tuning.

Findings

LARF effectively detects safety-degrading samples in datasets.

Removing identified samples mitigates safety degradation in LLMs.

LARF enhances safety alignment without compromising model performance.

Abstract

With rapid advancement and increasing accessibility of LLMs, fine-tuning aligned models has become a critical step for adapting them to real-world applications, which makes the safety of this fine-tuning process more important than ever. However, recent studies have highlighted a critical challenge: even when fine-tuning with seemingly benign downstream datasets, the safety of aligned LLMs can be compromised, making them more susceptible to malicious instructions. In this paper, we show that fine-tuning datasets often contain samples with safety-degrading features that are not easily identifiable on the surface. These samples can significantly degrade the safety alignment of LLMs during fine-tuning. To address this issue, we propose LARF, a Layer-Aware Representation Filtering method. This method identifies safety-sensitive layers within the LLM and leverages their representations to detect which data samples in the post-training dataset contain safety-degrading features. Experimental results demonstrate that LARF can effectively identify benign data with safety-degrading features. After removing such data, the safety alignment degradation caused by fine-tuning is mitigated. Please see our code at https://github.com/LLLeoLi/LARF.