AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basin

TL;DR

This paper introduces AsFT, a fine-tuning method that maintains LLM safety by constraining updates within a narrow safety basin, effectively reducing harmful behaviors while improving task performance.

Contribution

We propose AsFT, a novel fine-tuning approach that explicitly constrains update directions to preserve safety, addressing vulnerabilities caused by orthogonal perturbations.

Findings

AsFT reduces harmful behaviors by up to 7.60%.

AsFT improves task performance by 3.44%.

AsFT outperforms existing safety-preserving methods.

Abstract

Fine-tuning large language models (LLMs) improves performance but introduces critical safety vulnerabilities: even minimal harmful data can severely compromise safety measures. We observe that perturbations orthogonal to the alignment direction - defined by weight differences between aligned (safe) and unaligned models - rapidly compromise model safety. In contrast, updates along the alignment direction largely preserve it, revealing the parameter space as a "narrow safety basin". To address this, we propose AsFT (Anchoring Safety in Fine-Tuning) to maintain safety by explicitly constraining update directions during fine-tuning. By penalizing updates orthogonal to the alignment direction, AsFT effectively constrains the model within the "narrow safety basin," thus preserving its inherent safety. Extensive experiments on multiple datasets and models show that AsFT reduces harmful behaviors by up to 7.60%, improves task performance by 3.44%, and consistently outperforms existing methods across multiple tasks.