Do large language models solve verbal analogies like children do?

TL;DR

This study examines whether large language models solve verbal analogies through association like children, finding that they do, but their performance drops when controlling for associative processes, indicating reliance on association rather than relational mapping.

Contribution

The paper provides empirical evidence that LLMs solve verbal analogies primarily through association, similar to children, and introduces methods to distinguish associative from relational reasoning in models.

Findings

LLMs perform at levels comparable to children in solving analogies.

Controlling for association reduces models' performance by 1-2 years.

Associative processes are often responsible for correct analogy solutions.

Abstract

Analogy-making lies at the heart of human cognition. Adults solve analogies such as \textit{Horse belongs to stable like chicken belongs to ...?} by mapping relations (\textit{kept in}) and answering \textit{chicken coop}. In contrast, children often use association, e.g., answering \textit{egg}. This paper investigates whether large language models (LLMs) solve verbal analogies in A:B::C:? form using associations, similar to what children do. We use verbal analogies extracted from an online adaptive learning environment, where 14,002 7-12 year-olds from the Netherlands solved 622 analogies in Dutch. The six tested Dutch monolingual and multilingual LLMs performed around the same level as children, with MGPT performing worst, around the 7-year-old level, and XLM-V and GPT-3 the best, slightly above the 11-year-old level. However, when we control for associative processes this picture changes and each model's performance level drops 1-2 years. Further experiments demonstrate that associative processes often underlie correctly solved analogies. We conclude that the LLMs we tested indeed tend to solve verbal analogies by association with C like children do.