Attention as Binding: A Vector-Symbolic Perspective on Transformer Reasoning

TL;DR

This paper presents a vector-symbolic perspective on transformer models, interpreting their internal mechanisms as implementing approximate Vector Symbolic Architectures (VSA), which enhances understanding and guides architectural improvements for reasoning tasks.

Contribution

It introduces a unified algebraic framework for transformer attention, relates it to symbolic reasoning, and proposes VSA-inspired architectural biases and metrics for improved interpretability and robustness.

Findings

Transformer internals relate to chain-of-thought reasoning.

VSA-inspired biases improve symbolic manipulation.

Metrics for measuring VSA-likeness are proposed.

Abstract

Transformer-based language models display impressive reasoning-like behavior, yet remain brittle on tasks that require stable symbolic manipulation. This paper develops a unified perspective on these phenomena by interpreting self-attention and residual streams as implementing an approximate Vector Symbolic Architecture (VSA). In this view, queries and keys define role spaces, values encode fillers, attention weights perform soft unbinding, and residual connections realize superposition of many bound structures. We use this algebraic lens to relate transformer internals to chain-of-thought traces, program-based reasoning, and memory-augmented tool use, and to explain characteristic failure modes such as variable confusion and inconsistency across logically related prompts. Building on this perspective, we propose VSA-inspired architectural biases, including explicit binding/unbinding heads and hyperdimensional memory layers, and training objectives that promote role-filler separation and robust superposition. Finally, we outline metrics for measuring "VSA-likeness" and logical compositionality, and pose theoretical and architectural open problems. Overall, the paper argues that viewing attention as soft vector-symbolic computation offers a principled route toward more interpretable and logically reliable reasoning systems.