Not Yet AlphaFold for the Mind: Evaluating Centaur as a Synthetic Participant

TL;DR

This paper evaluates Centaur, a large language model fine-tuned on human data, for its potential as a human-like participant simulator in cognitive science, highlighting its strengths and current limitations.

Contribution

It provides a critical assessment of Centaur's capabilities as a participant simulator, identifying key gaps between its predictive accuracy and generative behavior.

Findings

Centaur predicts human behavior accurately in some tasks.

Its generative behavior diverges from actual human data.

It does not yet meet standards for reliable participant simulation.

Abstract

Simulators have revolutionized scientific practice across the natural sciences. By generating data that reliably approximate real-world phenomena, they enable scientists to accelerate hypothesis testing and optimize experimental designs. This is perhaps best illustrated by AlphaFold, a Nobel-prize winning simulator in chemistry that predicts protein structures from amino acid sequences, enabling rapid prototyping of molecular interactions, drug targets, and protein functions. In the behavioral sciences, a reliable participant simulator - a system capable of producing human-like behavior across cognitive tasks - would represent a similarly transformative advance. Recently, Binz et al. introduced Centaur, a large language model (LLM) fine-tuned on human data from 160 experiments, proposing its use not only as a model of cognition but also as a participant simulator for "in silico prototyping of experimental studies", e.g., to advance automated cognitive science. Here, we review the core criteria for a participant simulator and assess how well Centaur meets them. Although Centaur demonstrates strong predictive accuracy, its generative behavior - a critical criterion for a participant simulator - systematically diverges from human data. This suggests that, while Centaur is a significant step toward predicting human behavior, it does not yet meet the standards of a reliable participant simulator or an accurate model of cognition.