Pay What LLM Wants: Can LLM Simulate Economics Experiment with 522 Real-human Persona?

TL;DR

This study evaluates whether large language models can predict individual economic decisions using real human data from PWYW experiments, revealing limitations at the individual level but reasonable group-level behavioral replication.

Contribution

It is the first to systematically assess LLMs' ability to simulate real human economic decision-making with detailed personas in PWYW scenarios.

Findings

LLMs show limited accuracy in individual predictions

LLMs capture group-level behavioral tendencies reasonably well

Simple prompting methods perform nearly as well as advanced techniques

Abstract

Recent advances in Large Language Models (LLMs) have generated significant interest in their capacity to simulate human-like behaviors, yet most studies rely on fictional personas rather than actual human data. We address this limitation by evaluating LLMs' ability to predict individual economic decision-making using Pay-What-You-Want (PWYW) pricing experiments with real 522 human personas. Our study systematically compares three state-of-the-art multimodal LLMs using detailed persona information from 522 Korean participants in cultural consumption scenarios. We investigate whether LLMs can accurately replicate individual human choices and how persona injection methods affect prediction performance. Results reveal that while LLMs struggle with precise individual-level predictions, they demonstrate reasonable group-level behavioral tendencies. Also, we found that commonly adopted prompting techniques are not much better than naive prompting methods; reconstruction of personal narrative nor retrieval augmented generation have no significant gain against simple prompting method. We believe that these findings can provide the first comprehensive evaluation of LLMs' capabilities on simulating economic behavior using real human data, offering empirical guidance for persona-based simulation in computational social science.