Limited Ability of LLMs to Simulate Human Psychological Behaviours: a Psychometric Analysis

TL;DR

This study evaluates the capacity of GPT-3.5 and GPT-4 to simulate human psychological traits through standardized questionnaires, revealing GPT-4's limited success and questioning LLMs' ability to mimic individual human behaviors accurately.

Contribution

The paper applies psychometric analysis to assess LLMs' ability to simulate human psychological profiles, highlighting limitations especially with demographic-specific prompts.

Findings

GPT-4 shows some promising psychometric properties with generic personas.

Responses from both models with demographic profiles show poor psychometric validity.

Current LLMs are inadequate for accurately simulating individual human psychological traits.

Abstract

The humanlike responses of large language models (LLMs) have prompted social scientists to investigate whether LLMs can be used to simulate human participants in experiments, opinion polls and surveys. Of central interest in this line of research has been mapping out the psychological profiles of LLMs by prompting them to respond to standardized questionnaires. The conflicting findings of this research are unsurprising given that mapping out underlying, or latent, traits from LLMs' text responses to questionnaires is no easy task. To address this, we use psychometrics, the science of psychological measurement. In this study, we prompt OpenAI's flagship models, GPT-3.5 and GPT-4, to assume different personas and respond to a range of standardized measures of personality constructs. We used two kinds of persona descriptions: either generic (four or five random person descriptions) or specific (mostly demographics of actual humans from a large-scale human dataset). We found that the responses from GPT-4, but not GPT-3.5, using generic persona descriptions show promising, albeit not perfect, psychometric properties, similar to human norms, but the data from both LLMs when using specific demographic profiles, show poor psychometrics properties. We conclude that, currently, when LLMs are asked to simulate silicon personas, their responses are poor signals of potentially underlying latent traits. Thus, our work casts doubt on LLMs' ability to simulate individual-level human behaviour across multiple-choice question answering tasks.