The Neuroscience of Transformers

TL;DR

This paper explores how transformer models can serve as a computational analogy for cortical microcircuit organization, generating testable hypotheses about brain function and fostering cross-disciplinary insights.

Contribution

It proposes a hypothetical mapping between transformer operations and cortical features, bridging neuroscience and AI to enhance understanding of brain computation.

Findings

Predictions about laminar specialization and contextual modulation

Hypotheses on dendritic integration and oscillatory coordination

Framework for reciprocal insights between neuroscience and AI

Abstract

Neuroscience has long informed the development of artificial neural networks, but the success of modern architectures invites, in turn, the converse: can modern networks teach us lessons about brain function? Here, we examine the structure of the cortical column and propose that the transformer provides a natural computational analogy for multiple elements of cortical microcircuit organization. Rather than claiming a literal implementation of transformer equations in cortex, we develop a hypothetical mapping between transformer operations and laminar cortical features, using the analogy as an orienting framework for analysis and discussion. This mapping allows us to examine in greater depth how contextual selection, content routing, recurrent integration, and interlaminar transformations may be distributed across cortical circuitry. In doing so, we generate a broad set of predictions and experimentally testable hypotheses concerning laminar specialization, contextual modulation, dendritic integration, oscillatory coordination, and the effective connectivity of cortical columns. This proposal is intended as a structured hypothesis rather than a definitive account of cortical computation. Placing transformer operations and cortical architectonics into a common descriptive framework sharpens questions, reveals new functional correspondences, and opens a productive route for reciprocal exchange between systems neuroscience and modern AI. More broadly, this perspective suggests that comparing brains and architectures at the level of computational organization can yield genuine insight into both.