Can LLMs Assess Personality? Validating Conversational AI for Trait Profiling

TL;DR

This paper evaluates the potential of Large Language Models to assess personality traits through conversation, comparing their results with standard questionnaires and analyzing user perceptions of accuracy.

Contribution

It introduces a validated method for using LLMs in personality assessment, demonstrating their comparable accuracy and highlighting trait-specific calibration needs.

Findings

Moderate convergent validity with traditional questionnaires

Participants perceive LLM profiles as equally accurate

Trait-specific differences suggest calibration improvements

Abstract

This study validates Large Language Models (LLMs) as a dynamic alternative to questionnaire-based personality assessment. Using a within-subjects experiment (N=33), we compared Big Five personality scores derived from guided LLM conversations against the gold-standard IPIP-50 questionnaire, while also measuring user-perceived accuracy. Results indicate moderate convergent validity (r=0.38-0.58), with Conscientiousness, Openness, and Neuroticism scores statistically equivalent between methods. Agreeableness and Extraversion showed significant differences, suggesting trait-specific calibration is needed. Notably, participants rated LLM-generated profiles as equally accurate as traditional questionnaire results. These findings suggest conversational AI offers a promising new approach to traditional psychometrics.