Steering Vector Fields for Context-Aware Inference-Time Control in Large Language Models

TL;DR

This paper introduces Steering Vector Fields (SVF), a novel approach that learns context-dependent steering directions for large language models, significantly improving control reliability in various tasks and settings.

Contribution

SVF extends static steering vectors by modeling local, context-aware directions, enabling more reliable and flexible inference-time control in large language models.

Findings

SVF outperforms static SVs in control strength and reliability.

SVF maintains effectiveness in long-form and multi-attribute steering.

SVF enables coordinated multi-layer interventions for complex tasks.

Abstract

Steering vectors (SVs) offer a lightweight way to control large language models (LLMs) at inference time by shifting hidden activations, providing a practical middle ground between prompting and fine-tuning. Yet SVs can be unreliable in practice. Some concepts are unsteerable, and even when steering helps on average it can backfire for a non-trivial fraction of inputs. Reliability also degrades in long-form generation and multi-attribute steering. We take a geometric view of these failures. A static SV applies the same update vector everywhere in representation space, implicitly assuming that the concept-improving direction is constant across contexts. When the locally effective direction varies with the current activation, a single global vector can become misaligned, which yields weak or reversed effects. Guided by this perspective, we propose Steering Vector Fields (SVF), which learns a differentiable concept scoring function whose local gradient defines the steering direction at each activation, making interventions explicitly context-dependent. This formulation supports coordinated multi-layer interventions in a shared, aligned concept space, and enables efficient long-form and multi-attribute control within a unified framework. Across multiple LLMs and steering tasks, SVF delivers stronger and more reliable control, improving the practicality of inference-time steering.