From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directions

TL;DR

This paper introduces a method to control language model refusals by discovering and steering category-specific refusal directions in the model's internal representations, enhancing safety and reliability.

Contribution

It presents a novel approach to extract and utilize category-aligned refusal directions for inference-time control, improving safety without retraining.

Findings

Refusal token fine-tuning induces separable, category-aligned directions.

Constructed categorical steering vectors effectively control refusal behavior.

The low-rank combination method transfers across model variants, reducing over-refusals.

Abstract

Language models are commonly fine-tuned for safety alignment to refuse harmful prompts. One approach fine-tunes them to generate categorical refusal tokens that distinguish different refusal types before responding. In this work, we leverage a version of Llama 3 8B fine-tuned with these categorical refusal tokens to enable inference-time control over fine-grained refusal behavior, improving both safety and reliability. We show that refusal token fine-tuning induces separable, category-aligned directions in the residual stream, which we extract and use to construct categorical steering vectors with a lightweight probe that determines whether to steer toward or away from refusal during inference. In addition, we introduce a learned low-rank combination that mixes these category directions in a whitened, orthonormal steering basis, resulting in a single controllable intervention under activation-space anisotropy, and show that this intervention is transferable across same-architecture model variants without additional training. Across benchmarks, both categorical steering vectors and the low-rank combination consistently reduce over-refusals on benign prompts while increasing refusal rates on harmful prompts, highlighting their utility for multi-category refusal control.