Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondents

TL;DR

This study evaluates whether persona-conditioned large language models improve the reliability of synthetic survey responses, finding that such conditioning often introduces distortions and uneven effects across subgroups, challenging their current use.

Contribution

It provides a comprehensive empirical assessment of persona-conditioned LLMs in survey simulation, revealing potential drawbacks and heterogeneity in their effects.

Findings

Persona prompting does not consistently improve survey alignment.

Most items show minimal change, but some experience significant distortions.

Demographic conditioning can misallocate errors, affecting subgroup fidelity.

Abstract

Using persona-conditioned LLMs as synthetic survey respondents has become a common practice in computational social science and agent-based simulations. Yet, it remains unclear whether multi-attribute persona prompting improves LLM reliability or instead introduces distortions. Here we contribute to this assessment by leveraging a large dataset of U.S. microdata from the World Values Survey. Concretely, we evaluate two open-weight chat models and a random-guesser baseline across more than 70K respondent-item instances. We find that persona prompting does not yield a clear aggregate improvement in survey alignment and, in many cases, significantly degrades performance. Persona effects are highly heterogeneous as most items exhibit minimal change, while a small subset of questions and underrepresented subgroups experience disproportionate distortions. Our findings highlight a key adverse impact of current persona-based simulation practices: demographic conditioning can redistribute error in ways that undermine subgroup fidelity and risk misleading downstream analyses.