Thinking in Different Spaces: Domain-Specific Latent Geometry Survives Cross-Architecture Translation

TL;DR

This paper explores whether independently trained language models share compatible latent geometries and demonstrates that a linear projection can align their internal representations, enabling behavior correction without weight updates across diverse architectures and reasoning tasks.

Contribution

It introduces a method to align and intervene in different language models' internal states via linear projections, revealing domain-specific geometric properties and enabling cross-architecture behavioral correction.

Findings

Linear projections achieve moderate geometric alignment (R^2 ≈ 0.50) across heterogeneous models.

Behavioral correction rates range from 8.5% to 50% across reasoning tasks.

Projection matrices collapse when transferred across reasoning domains, indicating domain-specific geometry.

Abstract

We investigate whether independently trained language models converge to geometrically compatible latent representations, and whether this compatibility can be exploited to correct model behavior at inference time without any weight updates. We learn a linear projection matrix that maps activation vectors from a large teacher model into the coordinate system of a smaller student model, then intervene on the student's residual stream during generation by substituting its internal state with the translated teacher representation. Across a fully crossed experimental matrix of 20 heterogeneous teacher-student pairings spanning mixture-of-experts, dense, code-specialized, and synthetically trained architectures, the Ridge projection consistently achieves R^2 = 0.50 on verbal reasoning and R^2 = 0.40 on mathematical reasoning, collapsing to R^2 = -0.22 under permutation control and R^2 = 0.01 under L_1 regularization. Behavioral correction rates range from 14.0% to 50.0% on TruthfulQA (mean 25.2%) and from 8.5% to 43.3% on GSM8K arithmetic reasoning (mean 25.5%), demonstrating that the method generalizes across fundamentally different reasoning domains. We report a near-zero correlation between geometric alignment quality and behavioral correction rate (r = -0.07), revealing a dissociation between representation space fidelity and output space impact. Intervention strength is architecture-specific: student models exhibit characteristic sensitivity profiles that invert across domains, with the most steerable verbal student becoming the least steerable mathematical student. Finally, a double dissociation experiment conducted across all 20 model pairings confirms without exception that projection matrices collapse catastrophically when transferred across reasoning domains (mean R^2 = -3.83 in both transfer directions), establishing domain-specific subspace geometry as a universal property of LMs.