LLM-Mirror: A Generated-Persona Approach for Survey Pre-Testing

TL;DR

This paper introduces LLM-Mirror, a method using large language models with respondent-specific data to simulate individual survey responses, aiding in survey pre-testing and design optimization.

Contribution

It demonstrates that LLMs can replicate both distributional and individual-level survey responses when provided with respondent-specific information, advancing survey simulation techniques.

Findings

LLM-generated responses align with human responses in PLS-SEM analysis.

LLMs can reproduce individual human responses with respondent-specific data.

LLM-Mirror responses closely follow actual individual responses.

Abstract

Surveys are widely used in social sciences to understand human behavior, but their implementation often involves iterative adjustments that demand significant effort and resources. To this end, researchers have increasingly turned to large language models (LLMs) to simulate human behavior. While existing studies have focused on distributional similarities, individual-level comparisons remain underexplored. Building upon prior work, we investigate whether providing LLMs with respondents' prior information can replicate both statistical distributions and individual decision-making patterns using Partial Least Squares Structural Equation Modeling (PLS-SEM), a well-established causal analysis method. We also introduce the concept of the LLM-Mirror, user personas generated by supplying respondent-specific information to the LLM. By comparing responses generated by the LLM-Mirror with actual individual survey responses, we assess its effectiveness in replicating individual-level outcomes. Our findings show that: (1) PLS-SEM analysis shows LLM-generated responses align with human responses, (2) LLMs, when provided with respondent-specific information, are capable of reproducing individual human responses, and (3) LLM-Mirror responses closely follow human responses at the individual level. These findings highlight the potential of LLMs as a complementary tool for pre-testing surveys and optimizing research design.