Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space

TL;DR

This paper investigates the asymmetric effects of pretraining and alignment on transformer weights, revealing distinct geometric traces and their underlying causes through empirical and theoretical analysis.

Contribution

It characterizes the geometric asymmetry in transformer weights caused by pretraining and alignment, explaining it via anisotropic gradient accumulation and providing causal evidence.

Findings

Alignment updates concentrate in the read pathway ($W_Q$, $W_K$).

Pretraining induces prediction geometry in the write pathway ($W_O$, $W_2$).

Gradient anisotropy explains the observed weight-space patterns.

Abstract

Cross-entropy pretraining and preference alignment update the same transformer weights, but leave geometrically distinct traces. We characterise this asymmetry with a relative-subspace-fraction probe that tracks how weight deltas align with residual-stream activation subspaces and with the prediction subspace defined by the unembedding. Alignment deltas concentrate in the read pathway ($W_Q$, $W_K$), along principal directions of attention-input activations, while remaining near-isotropic in the write pathway ($W_O$, $W_2$) relative to the prediction subspace. We explain this pattern through anisotropic gradient accumulation: updates to a matrix $W$ are sums of outer products $\delta_t a_t^\top$, and inherit directional structure from whichever side has concentrated covariance. For read-pathway matrices, this side is the input activation $a_t$, whose covariance is spiked in trained transformers and therefore produces objective-agnostic concentration. For write-pathway matrices, the relevant side is the upstream gradient $\delta_t$, whose anisotropy depends on the loss. Cross-entropy supplies the canonical sharp per-sample signal, inducing write-pathway prediction geometry during pretraining; alignment objectives typically add little further write-side concentration. We support this explanation with a within-checkpoint trajectory, a graded contrastive-objective control, and a closed-form rank-1 intervention with matched direction controls, providing causal evidence for the proposed weight-space geometry.