Singular Vectors of Attention Heads Align with Features

TL;DR

This paper investigates when and why singular vectors of attention matrices in language models align with feature representations, providing theoretical justification and empirical evidence for their use in interpretability.

Contribution

It offers a theoretical framework and empirical validation for using singular vectors of attention matrices to identify features in language models.

Findings

Singular vectors align with features in observable models.

Theoretical conditions predict alignment in various scenarios.

Sparse attention decomposition indicates feature alignment in real models.

Abstract

Identifying feature representations in language models is a central task in mechanistic interpretability. Several recent studies have made an implicit assumption that feature representations can be inferred in some cases from singular vectors of attention matrices. However, sound justification for this assumption is lacking. In this paper we address that question, asking: why and when do singular vectors align with features? First, we demonstrate that singular vectors robustly align with features in a model where features can be directly observed. We then show theoretically that such alignment is expected under a range of conditions. We close by asking how, operationally, alignment may be recognized in real models where feature representations are not directly observable. We identify sparse attention decomposition as a testable prediction of alignment, and show evidence that it emerges in a manner consistent with predictions in real models. Together these results suggest that alignment of singular vectors with features can be a sound and theoretically justified basis for feature identification in language models.