Concept Generalization in Humans and Large Language Models: Insights from the Number Game

TL;DR

This study compares how humans and large language models generalize concepts in a number game, revealing fundamental differences in their inference strategies and sample efficiency, with humans showing more flexible and rapid generalization.

Contribution

The paper introduces a Bayesian framework to analyze and compare human and LLM concept inference, highlighting differences in biases and generalization capabilities.

Findings

Humans infer rule-based and similarity-based concepts flexibly.

LLMs rely more on mathematical rules for inference.

Humans generalize effectively from a single example.

Abstract

We compare human and large language model (LLM) generalization in the number game, a concept inference task. Using a Bayesian model as an analytical framework, we examined the inductive biases and inference strategies of humans and LLMs. The Bayesian model captured human behavior better than LLMs in that humans flexibly infer rule-based and similarity-based concepts, whereas LLMs rely more on mathematical rules. Humans also demonstrated a few-shot generalization, even from a single example, while LLMs required more samples to generalize. These contrasts highlight the fundamental differences in how humans and LLMs infer and generalize mathematical concepts.