Understanding Transformers through the Lens of Pavlovian Conditioning

TL;DR

This paper introduces a novel theoretical framework that interprets transformer attention mechanisms as Pavlovian conditioning, providing insights into their associative memory capabilities and architectural trade-offs.

Contribution

It reinterprets attention as an associative process akin to Pavlovian conditioning, offering new theoretical insights and capacity bounds for transformer models.

Findings

Attention can store O(√d_k) associations with error-free retrieval.

Average-case retrieval fidelity scales as O(d_k).

The framework reveals fundamental trade-offs in transformer architecture design.

Abstract

Transformer architectures have revolutionized artificial intelligence (AI) through their attention mechanisms, yet the computational principles underlying their success remain opaque. We present a novel theoretical framework that reinterprets the core computation of attention as Pavlovian conditioning. Our model finds a direct mathematical analogue in linear attention, which simplifies the analysis of the underlying associative process. We demonstrate that attention's queries, keys, and values can be mapped to the three elements of classical conditioning: test stimuli that probe associations, conditional stimuli (CS) that serve as retrieval cues, and unconditional stimuli (US) that contain response information. Through this lens, we suggest that each attention operation constructs a transient associative memory via a Hebbian rule, where CS-US pairs form dynamic associations that test stimuli can later retrieve. Our framework yields several theoretical insights grounded in this linearized model: (1) a capacity theorem showing that attention heads can store $O(\sqrt{d_k})$ associations for worst-case, error-free retrieval, while average-case retrieval fidelity scales robustly as $O(d_k)$; (2) an error propagation analysis revealing fundamental architectural trade-offs of balancing model depth, width, and head redundancy to maintain reliability; and (3) an understanding of how biologically plausible learning rules could enhance transformer architectures. By establishing this deep connection, we suggest that the success of modern AI may stem not from architectural novelty alone, but from implementing computational principles analogous to those optimized by biology over millions of years of evolution.