Can LLM "Self-report"?: Evaluating the Validity of Self-report Scales in Measuring Personality Design in LLM-based Chatbots

TL;DR

This study investigates whether large language model chatbots can reliably self-report their personality traits using questionnaires, finding weak correlations with human perceptions and interaction quality, thus questioning the validity of self-report methods.

Contribution

The paper critically evaluates the validity of self-report personality assessments in LLM-based chatbots and highlights the influence of context and interaction on these evaluations.

Findings

Weak correlation between self-reported and perceived personality traits

Self-report scores show limited predictive validity for interaction quality

Context and interaction significantly affect personality assessment accuracy

Abstract

A chatbot's personality design is key to interaction quality. As chatbots evolved from rule-based systems to those powered by large language models (LLMs), evaluating the effectiveness of their personality design has become increasingly complex, particularly due to the open-ended nature of interactions. A recent and widely adopted method for assessing the personality design of LLM-based chatbots is the use of self-report questionnaires. These questionnaires, often borrowed from established human personality inventories, ask the chatbot to rate itself on various personality traits. Can LLM-based chatbots meaningfully "self-report" their personality? We created 500 chatbots with distinct personality designs and evaluated the validity of their self-report personality scores by examining human perceptions formed during interactions with these chatbots. Our findings indicate that the chatbot's answers on human personality scales exhibit weak correlations with both human-perceived personality traits and the overall interaction quality. These findings raise concerns about both the criterion validity and the predictive validity of self-report methods in this context. Further analysis revealed the role of task context and interaction in the chatbot's personality design assessment. We further discuss design implications for creating more contextualized and interactive evaluation.