The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models

TL;DR

This paper systematically evaluates how different sociodemographic persona prompting strategies affect large language model simulations, revealing impacts on fidelity, stereotyping, and model performance across diverse groups.

Contribution

It provides a comprehensive analysis of prompt formulation effects on LLM sociodemographic simulations, offering practical guidance for improved prompt design.

Findings

Prompting style influences stereotyping and alignment.

Smaller models can outperform larger ones in sociodemographic tasks.

Interview-style prompts and name-based priming reduce stereotyping.

Abstract

Persona prompting is increasingly used in large language models (LLMs) to simulate views of various sociodemographic groups. However, how a persona prompt is formulated can significantly affect outcomes, raising concerns about the fidelity of such simulations. Using five open-source LLMs, we systematically examine how different persona prompt strategies, specifically role adoption formats and demographic priming strategies, influence LLM simulations across 15 intersectional demographic groups in both open- and closed-ended tasks. Our findings show that LLMs struggle to simulate marginalized groups but that the choice of demographic priming and role adoption strategy significantly impacts their portrayal. Specifically, we find that prompting in an interview-style format and name-based priming can help reduce stereotyping and improve alignment. Surprisingly, smaller models like OLMo-2-7B outperform larger ones such as Llama-3.3-70B. Our findings offer actionable guidance for designing sociodemographic persona prompts in LLM-based simulation studies.