Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence

TL;DR

Antibody is a novel defense method that mitigates harmful fine-tuning attacks on large language models by regularizing gradient influence of harmful samples, ensuring safety and performance during fine-tuning.

Contribution

The paper introduces Antibody, a two-stage strategy combining safety alignment and gradient regularization to defend against harmful fine-tuning attacks on large language models.

Findings

Effective mitigation of harmful fine-tuning attacks.

Improved safety and robustness of models during fine-tuning.

Enhanced performance on user-submitted datasets.

Abstract

Fine-tuning-as-a-service introduces a threat to Large Language Models' safety when service providers fine-tune their models on poisoned user-submitted datasets, a process known as harmful fine-tuning attacks. In this work, we show that by regularizing the gradient contribution of harmful samples encountered during fine-tuning, we can effectively mitigate the impact of harmful fine-tuning attacks. To this end, we introduce Antibody, a defense strategy that first ensures robust safety alignment for the model before fine-tuning, and then applies a safety-preservation learning algorithm during fine-tuning. Specifically, in the alignment stage before fine-tuning, we propose optimizing the model to be in a flat loss region with respect to harmful samples, which makes the safety alignment more resilient to subsequent harmful fine-tuning. Then, in the fine-tuning stage, we design a fine-tuning algorithm that applies a weighting scheme to all samples in each training batch to inhibit the model from learning from harmful samples while encouraging learning from benign samples. Experimental results demonstrate that Antibody successfully mitigates harmful fine-tuning attacks while boosting fine-tuning performance on the user-submitted dataset.