Mind Reading or Misreading? LLMs on the Big Five Personality Test

TL;DR

This study evaluates large language models for automatic personality prediction using the Big Five model, revealing variability in performance, challenges in reliability, and the importance of prompt design and evaluation metrics.

Contribution

It systematically compares multiple LLMs and prompting strategies for personality prediction, highlighting current limitations and guiding future improvements.

Findings

Enriched prompts improve output validity but introduce bias.

Openness and Agreeableness are easier to predict than Extraversion and Neuroticism.

No zero-shot configuration achieves consistent reliability across traits.

Abstract

We evaluate large language models (LLMs) for automatic personality prediction from text under the binary Five Factor Model (BIG5). Five models -- including GPT-4 and lightweight open-source alternatives -- are tested across three heterogeneous datasets (Essays, MyPersonality, Pandora) and two prompting strategies (minimal vs. enriched with linguistic and psychological cues). Enriched prompts reduce invalid outputs and improve class balance, but also introduce a systematic bias toward predicting trait presence. Performance varies substantially: Openness and Agreeableness are relatively easier to detect, while Extraversion and Neuroticism remain challenging. Although open-source models sometimes approach GPT-4 and prior benchmarks, no configuration yields consistently reliable predictions in zero-shot binary settings. Moreover, aggregate metrics such as accuracy and macro-F1 mask significant asymmetries, with per-class recall offering clearer diagnostic value. These findings show that current out-of-the-box LLMs are not yet suitable for APPT, and that careful coordination of prompt design, trait framing, and evaluation metrics is essential for interpretable results.