TopoLM: brain-like spatio-functional organization in a topographic language model
Neil Rathi, Johannes Mehrer, Badr AlKhamissi, Taha Binhuraib, Nicholas M. Blauch, Martin Schrimpf
TL;DR
TopoLM introduces a spatially organized transformer model that mimics brain-like clustering of language functions, revealing that spatial objectives can explain neural organization in the human brain.
Contribution
This work develops a topographically organized transformer model with a spatial loss, aligning artificial representations with brain language organization.
Findings
Model clusters correspond to semantic categories
Replicates cortical language organization
Supports spatial objectives as a basis for neural organization
Abstract
Neurons in the brain are spatially organized such that neighbors on tissue often exhibit similar response profiles. In the human language system, experimental studies have observed clusters for syntactic and semantic categories, but the mechanisms underlying this functional organization remain unclear. Here, building on work from the vision literature, we develop TopoLM, a transformer language model with an explicit two-dimensional spatial representation of model units. By combining a next-token prediction objective with a spatial smoothness loss, representations in this model assemble into clusters that correspond to semantically interpretable groupings of text and closely match the functional organization in the brain's language system. TopoLM successfully predicts the emergence of the spatio-functional organization of a cortical language system as well as the organization of…
Peer Reviews
Decision·ICLR 2025 Oral
The authors contextualized and described with clarity the strategy introduced to incorporate spatial information of activations in transformers for natural language processing. Both the training details and experiments are very well explained and organized. The work, inspired by previous efforts in vision models, introduces for the first time (to the reviewer knowledge) explicit spatial information in the loss of transformers trained on natural language.
Given the strong inspiration from human neural activity studies of the work, the work would benefit from more extended discussion and analyses related to some architectural choices (e.g. random permutation of spatial position between layers and role of multi-head attention).
TopoLM introduces a novel approach to understanding the spatial organization of language processing in the human brain. By integrating two-dimensional spatial encoding into a transformer architecture, the model effectively captures the topological structure of language, which has not been extensively explored in existing literature. This innovative perspective opens new avenues for research in language modeling and cognitive neuroscience. Meanwhile, this research has significant implications
While the paper presents a promising theoretical framework, the empirical evaluation of TopoLM is somewhat limited. The experiments primarily focus on a small set of tasks, which may not fully capture the model's robustness across diverse linguistic contexts. Meanwhile, this paper does not sufficiently explore or compare TopoLM with other existing topological models in language processing. Including a comparative analysis with relevant models could strengthen the argument for TopoLM's effecti
The paper addresses a novel and important issue. There has been little work on training LLMs with topography constraints. Successful training opens the door to interesting analyses, and potentially more flexible approaches for supervised learning of model-to-brain alignment. The writing is clear, and the analyses are interesting in that the authors apply well established statistical approaches used in the human neuroimaging literature to the model itself.
**P.S following the discussion period: I believe the more critical perspective that was adopted by the authors during the review process improves the soundness of the paper. I have modified my evaluation.** 1. The spatial loss term is a key element in the work and should be better motivated. 1a. As phrased in the paper, the loss term constrains neurons that are closer to each other (according to a predefined spatial arrangement) to have more strongly correlated activation profiles than those
Code & Models
Videos
Taxonomy
TopicsSpeech and dialogue systems · Cognitive Science and Mapping