A Nurse is Blue and Elephant is Rugby: Cross Domain Alignment in Large Language Models Reveal Human-like Patterns

TL;DR

This study evaluates large language models' ability to perform cross-domain concept mappings, revealing that models exhibit human-like patterns in their representations and reasoning processes, suggesting similarities to human cognition.

Contribution

The paper introduces a novel cognitive science-inspired task to assess LLMs' conceptualization and reasoning, demonstrating their human-like mapping and explanation behaviors.

Findings

Models show human-like concept mappings.

Models generate valid, human-like explanations.

Behavioral patterns in models resemble human reasoning.

Abstract

Cross-domain alignment refers to the task of mapping a concept from one domain to another. For example, ``If a \textit{doctor} were a \textit{color}, what color would it be?''. This seemingly peculiar task is designed to investigate how people represent concrete and abstract concepts through their mappings between categories and their reasoning processes over those mappings. In this paper, we adapt this task from cognitive science to evaluate the conceptualization and reasoning abilities of large language models (LLMs) through a behavioral study. We examine several LLMs by prompting them with a cross-domain mapping task and analyzing their responses at both the population and individual levels. Additionally, we assess the models' ability to reason about their predictions by analyzing and categorizing their explanations for these mappings. The results reveal several similarities between humans' and models' mappings and explanations, suggesting that models represent concepts similarly to humans. This similarity is evident not only in the model representation but also in their behavior. Furthermore, the models mostly provide valid explanations and deploy reasoning paths that are similar to those of humans.