LLM Personas as a Substitute for Field Experiments in Method Benchmarking

TL;DR

This paper demonstrates that under certain conditions, LLM-based personas can replace human field experiments for method benchmarking, providing a cost-effective and statistically reliable alternative.

Contribution

It provides a theoretical characterization of when persona-based benchmarking is equivalent to field experiments and offers explicit sample size bounds for reliable method discrimination.

Findings

Persona substitution is valid under aggregate-only and method-blind evaluation conditions.

Sample size bounds are derived for reliable method discrimination using personas.

The approach offers a cost-effective alternative to traditional field experiments.

Abstract

Field experiments (A/B tests) are often the most credible benchmark for methods (algorithms) in societal systems, but their cost and latency bottleneck rapid methodological progress. LLM-based persona simulation offers a cheap synthetic alternative, yet it is unclear whether replacing humans with personas preserves the benchmark interface that adaptive methods optimize against. We prove an if-and-only-if characterization: when (i) methods observe only the aggregate outcome (aggregate-only observation) and (ii) evaluation depends only on the submitted artifact and not on the method's identity or provenance (method-blind evaluation), swapping humans for personas is just panel change from the method's point of view, indistinguishable from changing the evaluation population (e.g., New York to Jakarta). Furthermore, we move from validity to usefulness: we define an information-theoretic discriminability of the induced aggregate channel and show that making persona benchmarking as decision-relevant as a field experiment is fundamentally a sample-size question, yielding explicit bounds on the number of independent persona evaluations required to reliably distinguish meaningfully different methods at a chosen resolution.