From Data Statistics to Feature Geometry: How Correlations Shape Superposition

TL;DR

This paper introduces BOWS, a controlled setting demonstrating that feature correlations in neural networks can lead to constructive interference, revealing new geometric structures and semantic clustering in language models.

Contribution

It extends the understanding of superposition by showing how correlated features can produce constructive interference and semantic structures, unlike the idealized uncorrelated case.

Findings

Correlated features can cause constructive interference in superposition.

Semantic clusters and cyclical structures emerge in language models.

Weight decay influences feature arrangements and correlations.

Abstract

A central idea in mechanistic interpretability is that neural networks represent more features than they have dimensions, arranging them in superposition to form an over-complete basis. This framing has been influential, motivating dictionary learning approaches such as sparse autoencoders. However, superposition has mostly been studied in idealized settings where features are sparse and uncorrelated. In these settings, superposition is typically understood as introducing interference that must be minimized geometrically and filtered out by non-linearities such as ReLUs, yielding local structures like regular polytopes. We show that this account is incomplete for realistic data by introducing Bag-of-Words Superposition (BOWS), a controlled setting to encode binary bag-of-words representations of internet text in superposition. Using BOWS, we find that when features are correlated, interference can be constructive rather than just noise to be filtered out. This is achieved by arranging features according to their co-activation patterns, making interference between active features constructive, while still using ReLUs to avoid false positives. We show that this kind of arrangement is more prevalent in models trained with weight decay and naturally gives rise to semantic clusters and cyclical structures which have been observed in real language models yet were not explained by the standard picture of superposition. Code for this paper can be found at https://github.com/LucasPrietoAl/correlations-feature-geometry.