LLMs as Models for Analogical Reasoning

TL;DR

This paper investigates the ability of large language models to perform analogical reasoning on novel tasks involving semantic and abstract content, comparing their performance to humans and exploring the implications for cognitive modeling.

Contribution

It introduces new analogical reasoning tasks that challenge LLMs to re-represent semantic information, revealing their capabilities and limitations in modeling human-like analogy.

Findings

LLMs match human performance on several reasoning conditions

Humans and LLMs respond differently to task variations

Results suggest LLMs offer a how-possibly but not a how-actually explanation of human analogy

Abstract

Analogical reasoning -- the capacity to identify and map structural relationships between different domains -- is fundamental to human cognition and learning. Recent studies have shown that large language models (LLMs) can sometimes match humans in analogical reasoning tasks, opening the possibility that analogical reasoning might emerge from domain-general processes. However, it is still debated whether these emergent capacities are largely superficial and limited to simple relations seen during training or whether they encompass the flexible representational and mapping capabilities which are the focus of leading cognitive models of analogy. In this study, we introduce novel analogical reasoning tasks that require participants to map between semantically contentful words and sequences of letters and other abstract characters. This task necessitates the ability to flexibly re-represent rich semantic information -- an ability which is known to be central to human analogy but which is thus far not well captured by existing cognitive theories and models. We assess the performance of both human participants and LLMs on tasks focusing on reasoning from semantic structure and semantic content, introducing variations that test the robustness of their analogical inferences. Advanced LLMs match human performance across several conditions, though humans and LLMs respond differently to certain task variations and semantic distractors. Our results thus provide new evidence that LLMs might offer a how-possibly explanation of human analogical reasoning in contexts that are not yet well modeled by existing theories, but that even today's best models are unlikely to yield how-actually explanations.