Evaluating LLMs as Human Surrogates in Controlled Experiments

TL;DR

This study assesses whether large language models can reliably simulate human responses in behavioral experiments by comparing their outputs to actual human data.

Contribution

It provides a systematic evaluation of LLMs as surrogates for humans in controlled survey experiments, highlighting their capabilities and limitations.

Findings

LLMs reproduce some directional effects seen in humans.

Effect magnitudes and moderation patterns vary across models.

LLMs capture aggregate belief-updating but not all human-scale effects.

Abstract

Large language models (LLMs) are increasingly used to simulate human responses in behavioral research, yet it remains unclear when LLM-generated data support the same experimental inferences as human data. We evaluate this by directly comparing off-the-shelf LLM-generated responses with human responses from a canonical survey experiment on accuracy perception. Each human observation is converted into a structured prompt, and models generate a single 0--10 outcome variable without task-specific training; identical statistical analyses are applied to human and synthetic responses. We find that LLMs reproduce several directional effects observed in humans, but effect magnitudes and moderation patterns vary across models. Off-the-shelf LLMs therefore capture aggregate belief-updating patterns under controlled conditions but do not consistently match human-scale effects, clarifying when LLM-generated data can function as behavioral surrogates.