Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence

TL;DR

This paper introduces activation steering methods to mitigate misalignment in large language models during generation, improving safety traits like honesty and compassion without sacrificing coherence.

Contribution

It proposes novel projection-aware activation steering techniques that selectively intervene based on activation thresholds, enhancing safety while maintaining model capabilities.

Findings

All methods improve honesty and compassion in generated outputs.

StTP and StMP better preserve general capabilities and reduce repetition.

Activation steering effectively mitigates misalignment in LLMs during runtime.

Abstract

Alignment in LLMs is more brittle than commonly assumed: misalignment can be triggered by adversarial prompts, benign fine-tuning, emergent misalignment, and goal misgeneralization. Recent evidence suggests that some misalignment behaviors are encoded as linear structure in activation space, making it tractable via steering, while safety alignment has been shown to govern the first few output tokens primarily, leaving subsequent generation unguarded. These findings motivate activation steering as a lightweight runtime defense that continuously corrects misaligned activations throughout generation. We evaluate three methods: Steer-With-Fixed-Coeff (SwFC), which applies uniform additive steering, and two novel projection-aware methods, Steer-to-Target-Projection (StTP) and Steer-to-Mirror-Projection (StMP), that use a logistic regression decision boundary to selectively intervene only on tokens whose activations fall below distributional thresholds. Using malicious system prompts as a controlled proxy for misalignment, we evaluate under two threat models (dishonesty and dismissiveness) and two architectures (Llama-3.3-70B-Instruct, Qwen3-32B). All methods substantially recover target traits (honesty and compassion) while preserving coherence. StTP and StMP better maintain general capabilities (MMLU, MT-Bench, AlpacaEval) and produce less repetition in multi-turn conversations.