LLMs Can Unlearn Refusal with Only 1,000 Benign Samples

TL;DR

This paper uncovers a vulnerability in LLM safety alignment where models can unlearn refusal behaviors with minimal fine-tuning on benign samples, challenging assumptions about safety mechanisms.

Contribution

It introduces a novel refusal unlearning technique that effectively reduces LLMs' refusal responses using only 1,000 benign samples, supported by theoretical proofs.

Findings

Refusal unlearning degrades safety scores across 16 LLMs

The method is effective on both open-source and closed-source models

Results are not due to simple fine-tuning or prefix effects

Abstract

This study reveals a previously unexplored vulnerability in the safety alignment of Large Language Models (LLMs). Existing aligned LLMs predominantly respond to unsafe queries with refusals, which often begin with a fixed set of prefixes (I'm sorry). We demonstrate that this rigid refusal pattern is a vulnerability and introduce a novel \textbf{refusal unlearning} technique that exploits it. Specifically, we fine-tune LLMs using merely 1,000 benign samples, where each response is prepended with a refusal prefix. The underlying intuition is to disrupt the refusal completion pathway, thereby driving the model to forget how to refuse while following harmful instructions. This intuition is further supported by theoretical proofs. We apply this approach to a total of 16 LLMs, including various open-source models from Llama, Qwen, and Gemma families, as well as closed-source models such as Gemini and GPT. Experimental results show that the safety scores of previously aligned LLMs degrade both consistently and substantially. Importantly, we verify that the observed gain cannot be attributed to plain fine-tuning or random prefix effects. Our findings suggest that current safety alignment may rely heavily on token sequence memorization rather than reasoning, motivating future work beyond simple refusal mechanisms. Code has been released: https://github.com/guoyang9/refusal-unlearning.