Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency

TL;DR

This paper reviews 63 studies on synthetic personae in LLM research, highlighting gaps in representativeness and proposing a transparency checklist to improve ecological validity and methodological rigor.

Contribution

It provides a comprehensive assessment of current persona-based experiments and introduces practical guidelines for enhancing transparency and validity in LLM alignment studies.

Findings

Most studies focus on limited sociodemographic attributes.

Only 35% discuss the representativeness of their LLM personae.

Significant variability exists in how experiments specify tasks and populations.

Abstract

Synthetic personae experiments have become a prominent method in Large Language Model alignment research, yet the representativeness and ecological validity of these personae vary considerably between studies. Through a review of 63 peer-reviewed studies published between 2023 and 2025 in leading NLP and AI venues, we reveal a critical gap: task and population of interest are often underspecified in persona-based experiments, despite personalization being fundamentally dependent on these criteria. Our analysis shows substantial differences in user representation, with most studies focusing on limited sociodemographic attributes and only 35% discussing the representativeness of their LLM personae. Based on our findings, we introduce a persona transparency checklist that emphasizes representative sampling, explicit grounding in empirical data, and enhanced ecological validity. Our work provides both a comprehensive assessment of current practices and practical guidelines to improve the rigor and ecological validity of persona-based evaluations in language model alignment research.