Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas

TL;DR

This study explores how explicitly and implicitly prompting LLMs with human-like personas affects their ability to reflect human diversity and biases in social science tasks, revealing limitations in modeling complex human perceptions.

Contribution

It introduces a method to evaluate LLMs with human personas and compares explicit versus implicit prompting, highlighting their strengths and limitations in capturing human biases.

Findings

Explicit personas sometimes reproduce human biases.

LLMs generally fail to demonstrate implicit biases.

Models capture statistical speech patterns but not complex perceptions.

Abstract

Large language models (LLMs) are increasingly being used in human-centered social scientific tasks, such as data annotation, synthetic data creation, and engaging in dialog. However, these tasks are highly subjective and dependent on human factors, such as one's environment, attitudes, beliefs, and lived experiences. Thus, it may be the case that employing LLMs (which do not have such human factors) in these tasks results in a lack of variation in data, failing to reflect the diversity of human experiences. In this paper, we examine the role of prompting LLMs with human-like personas and asking the models to answer as if they were a specific human. This is done explicitly, with exact demographics, political beliefs, and lived experiences, or implicitly via names prevalent in specific populations. The LLM personas are then evaluated via (1) subjective annotation task (e.g., detecting toxicity) and (2) a belief generation task, where both tasks are known to vary across human factors. We examine the impact of explicit vs. implicit personas and investigate which human factors LLMs recognize and respond to. Results show that explicit LLM personas show mixed results when reproducing known human biases, but generally fail to demonstrate implicit biases. We conclude that LLMs may capture the statistical patterns of how people speak, but are generally unable to model the complex interactions and subtleties of human perceptions, potentially limiting their effectiveness in social science applications.