Emergent Analogical Reasoning in Transformers

TL;DR

This paper investigates how Transformers develop analogical reasoning, revealing that it emerges through geometric alignment and functor application, influenced by data, optimization, and scale, and is observable in pretrained models.

Contribution

It formalizes analogical reasoning in Transformers using category theory concepts and introduces synthetic tasks to evaluate its emergence and mechanisms.

Findings

Analogical reasoning emergence is sensitive to data and model scale.

Mechanistic analysis shows geometric alignment and functor application are key.

Pretrained LLMs also exhibit these analogical reasoning mechanisms.

Abstract

Analogy is a central faculty of human intelligence, enabling abstract patterns discovered in one domain to be applied to another. Despite its central role in cognition, the mechanisms by which Transformers acquire and implement analogical reasoning remain poorly understood. In this work, inspired by the notion of functors in category theory, we formalize analogical reasoning as the inference of correspondences between entities across categories. Based on this formulation, we introduce synthetic tasks that evaluate the emergence of analogical reasoning under controlled settings. We find that the emergence of analogical reasoning is highly sensitive to data characteristics, optimization choices, and model scale. Through mechanistic analysis, we show that analogical reasoning in Transformers decomposes into two key components: (1) geometric alignment of relational structure in the embedding space, and (2) the application of a functor within the Transformer. These mechanisms enable models to transfer relational structure from one category to another, realizing analogy. Finally, we quantify these effects and find that the same trends are observed in pretrained LLMs. In doing so, we move analogy from an abstract cognitive notion to a concrete, mechanistically grounded phenomenon in modern neural networks.