Fine-Grained Activation Steering: Steering Less, Achieving More

TL;DR

This paper introduces AUSteer, a fine-grained activation steering method for large language models that improves efficiency and effectiveness by targeting atomic units rather than entire blocks, leading to better control with less intervention.

Contribution

The paper reveals the heterogeneity of block activations and proposes a novel AU-level steering method, AUSteer, which enhances precision and reduces unnecessary modifications in LLM behavior.

Findings

AUSteer outperforms existing methods across multiple tasks and models.

Steering at the AU level requires fewer activations while achieving better control.

Heterogeneity in block activations causes coarse steering and is mitigated by fine-grained intervention.

Abstract

Activation steering has emerged as a cost-effective paradigm for modifying large language model (LLM) behaviors. Existing methods typically intervene at the block level, steering the bundled activations of selected attention heads, feedforward networks, or residual streams. However, we reveal that block-level activations are inherently heterogeneous, entangling beneficial, irrelevant, and harmful features, thereby rendering block-level steering coarse, inefficient, and intrusive. To investigate the root cause, we decompose block activations into fine-grained atomic unit (AU)-level activations, where each AU-level activation corresponds to a single dimension of the block activation, and each AU denotes a slice of the block weight matrix. Steering an AU-level activation is thus equivalent to steering its associated AU. Our theoretical and empirical analysis show that heterogeneity arises because different AUs or dimensions control distinct token distributions in LLM outputs. Hence, block-level steering inevitably moves helpful and harmful token directions together, which reduces efficiency. Restricting intervention to beneficial AUs yields more precise and effective steering. Building on this insight, we propose AUSteer, a simple and efficient method that operates at a finer granularity of the AU level. AUSteer first identifies discriminative AUs globally by computing activation momenta on contrastive samples. It then assigns adaptive steering strengths tailored to diverse inputs and selected AU activations. Comprehensive experiments on multiple LLMs and tasks show that AUSteer consistently surpasses advanced baselines while steering considerably fewer activations, demonstrating that steering less achieves more.