D-STEER - Preference Alignment Techniques Learn to Behave, not to Believe -- Beneath the Surface, DPO as Steering Vector Perturbation in Activation Space

TL;DR

This paper reveals that DPO aligns language models by steering activations along a few preference directions rather than rewriting beliefs, acting as a low-rank adjustment in activation space that influences behavior.

Contribution

It introduces a simple derivation showing DPO's effect as a low-rank steering mechanism and demonstrates how this explains the alignment process in language models.

Findings

DPO induces a first-order shift in hidden representations.

Empirical steering vectors can reproduce aligned behavior.

Spectral analysis shows rank-one dominance and entropy collapse.

Abstract

Direct Preference Optimization (DPO) has become a standard recipe for aligning large language models, yet it is still unclear what kind of change it actually induces inside the network. This paper argues that DPO does not rewrite a models internal beliefs; instead, it acts as a low rank steering mechanism that nudges activations along a small number of preference directions. Using a simple derivation, we show that the DPO gradient depends only on the difference between the logit embeddings of preferred and dispreferred completions, implying a first order shift in the final hidden representation rather than a deep restructuring of semantics. We then extract an empirical steering vector from a DPO tuned model and demonstrate that adding this vector to base activations reproduces most of the aligned behavior, while subtracting it nearly restores the original model. Finally, spectral analyses reveal rank-one dominance and entropy collapse in upper layers, indicating that alignment is funneled through a narrow subspace. Taken together, these results support a behavioral illusion view of DPO: it teaches models how to act aligned, not what to believe.