Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics

TL;DR

This paper introduces TRAIT, a new benchmark for assessing LLMs' personalities using psychometrically validated questionnaires, revealing that LLMs have distinct, consistent personalities influenced by training data, with current prompts limited in eliciting certain traits.

Contribution

The paper presents TRAIT, a novel, reliable benchmark for measuring LLM personalities based on validated psychometric tests, and demonstrates their personality traits are influenced by training data.

Findings

LLMs exhibit distinct and consistent personalities.

Training data significantly influences LLM personality traits.

Current prompting techniques are limited in eliciting certain personality traits.

Abstract

Recent advancements in Large Language Models (LLMs) have led to their adaptation in various domains as conversational agents. We wonder: can personality tests be applied to these agents to analyze their behavior, similar to humans? We introduce TRAIT, a new benchmark consisting of 8K multi-choice questions designed to assess the personality of LLMs. TRAIT is built on two psychometrically validated small human questionnaires, Big Five Inventory (BFI) and Short Dark Triad (SD-3), enhanced with the ATOMIC-10X knowledge graph to a variety of real-world scenarios. TRAIT also outperforms existing personality tests for LLMs in terms of reliability and validity, achieving the highest scores across four key metrics: Content Validity, Internal Validity, Refusal Rate, and Reliability. Using TRAIT, we reveal two notable insights into personalities of LLMs: 1) LLMs exhibit distinct and consistent personality, which is highly influenced by their training data (e.g., data used for alignment tuning), and 2) current prompting techniques have limited effectiveness in eliciting certain traits, such as high psychopathy or low conscientiousness, suggesting the need for further research in this direction.