Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History

TL;DR

This study reveals persistent instability in LLMs' personality assessments, showing that model size, reasoning, and conversation history have limited impact on stability, raising concerns about their reliability for safety-critical uses.

Contribution

Introduces PERSIST, a comprehensive framework for evaluating personality stability in open-source LLMs, highlighting fundamental limitations in achieving consistent behavioral traits.

Findings

Question reordering causes large personality shifts.

Scaling offers limited stability improvements.

Interventions like reasoning and history can increase variability.

Abstract

Large language models require consistent behavioral patterns for safe deployment, yet there are indications of large variability that may lead to an instable expression of personality traits in these models. We present PERSIST (PERsonality Stability in Synthetic Text), a comprehensive evaluation framework testing 25 open-source models (1B-685B parameters) across 2 million+ responses. Using traditional (BFI, SD3) and novel LLM-adapted personality questionnaires, we systematically vary model size, personas, reasoning modes, question order or paraphrasing, and conversation history. Our findings challenge fundamental assumptions: (1) Question reordering alone can introduce large shifts in personality measurements; (2) Scaling provides limited stability gains: even 400B+ models exhibit standard deviations >0.3 on 5-point scales; (3) Interventions expected to stabilize behavior, such as reasoning and inclusion of conversation history, can paradoxically increase variability; (4) Detailed persona instructions produce mixed effects, with misaligned personas showing significantly higher variability than the helpful assistant baseline; (5) The LLM-adapted questionnaires, despite their improved ecological validity, exhibit instability comparable to human-centric versions. This persistent instability across scales and mitigation strategies suggests that current LLMs lack the architectural foundations for genuine behavioral consistency. For safety-critical applications requiring predictable behavior, these findings indicate that current alignment strategies may be inadequate.