Dissociating Decodability and Causal Use in Bracket-Sequence Transformers

TL;DR

This paper investigates whether hierarchical representations in transformers are causally used or merely decodable, revealing that decodability does not necessarily imply causal influence.

Contribution

It demonstrates through probing and intervention that certain hierarchical signals are decodable but not causally necessary for transformer performance.

Findings

Masking attention to the top-of-stack reduces long-distance accuracy.

Ablating residual stream subspaces has little effect on performance.

Hierarchical signals are decodable but not always causally used.

Abstract

When trained on tasks requiring an understanding of hierarchical structure, transformers have been found to represent this hierarchy in distinct ways: in the geometry of the residual stream, and in stack-like attention patterns maintaining a last-in, first-out ordering. However, it remains unclear whether these representations are causally used or merely decodable. We examine this gap in transformers trained on the Dyck language (a formal language of balanced bracket sequences), where the hierarchical ground truth is explicit. By probing and intervening on the residual stream and attention patterns, we find that depth, distance, and top-of-stack signals are all decodable, yet their causal roles diverge. Specifically, masking attention to the true top-of-stack position causes a sharp drop in long-distance accuracy, while ablating low-dimensional residual stream subspaces has comparatively little effect. These results, which extend to a templated natural language setting, suggest that even in a controlled setting where the relevant hierarchical variables are known, decodability alone does not imply causal use.