Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sink

TL;DR

This paper introduces Surgery, a method that uses attention sink mechanisms to mitigate harmful fine-tuning in large language models, improving safety performance across multiple benchmarks.

Contribution

It proposes a novel attention sink-based regularizer and the separable sink divergence hypothesis to effectively reduce harmful pattern learning during fine-tuning.

Findings

Surgery improves safety benchmark scores by up to 11.25%.

Attention heads with positive sink divergence correlate with increased harmfulness.

The sink divergence regularizer effectively steers attention heads away from harmful patterns.

Abstract

Harmful fine-tuning can invalidate safety alignment of large language models, exposing significant safety risks. In this paper, we utilize the attention sink mechanism to mitigate harmful fine-tuning. Specifically, we first measure a statistic named \emph{sink divergence} for each attention head and observe that \emph{different attention heads exhibit two different signs of sink divergence}. To understand its safety implications, we conduct experiments and find that the number of attention heads of positive sink divergence increases along with the increase of the model's harmfulness when undergoing harmful fine-tuning. Based on this finding, we propose a separable sink divergence hypothesis -- \emph{attention heads associating with learning harmful patterns during fine-tuning are separable by their sign of sink divergence}. Based on the hypothesis, we propose a fine-tuning-stage defense, dubbed Surgery. Surgery utilizes a regularizer for sink divergence suppression, which steers attention heads toward the negative sink divergence group, thereby reducing the model's tendency to learn and amplify harmful patterns. Extensive experiments demonstrate that Surgery improves defense performance by 5.90\%, 11.25\%, and 9.55\% on the BeaverTails, HarmBench, and SorryBench benchmarks, respectively. Source code is available on https://github.com/Lslland/Surgery.