InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews
Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo,, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, Yanghua, Xiao

TL;DR
This paper introduces InCharacter, a novel method using psychological interviews to evaluate the personality fidelity of role-playing agents, demonstrating high alignment with human perceptions across multiple characters.
Contribution
The paper presents InCharacter, a new evaluation framework employing psychological scales to assess personality fidelity in RPAs, addressing limitations of previous methods.
Findings
InCharacter effectively measures RPA personalities with high validity.
State-of-the-art RPAs achieve up to 80.7% accuracy in personality alignment.
The method works across diverse characters and psychological scales.
Abstract
Role-playing agents (RPAs), powered by large language models, have emerged as a flourishing field of applications. However, a key challenge lies in assessing whether RPAs accurately reproduce the personas of target characters, namely their character fidelity. Existing methods mainly focus on the knowledge and linguistic patterns of characters. This paper, instead, introduces a novel perspective to evaluate the personality fidelity of RPAs with psychological scales. Overcoming drawbacks of previous self-report assessments on RPAs, we propose InCharacter, namely Interviewing Character agents for personality tests. Experiments include various types of RPAs and LLMs, covering 32 distinct characters on 14 widely used psychological scales. The results validate the effectiveness of InCharacter in measuring RPA personalities. Then, with InCharacter, we show that state-of-the-art RPAs exhibit…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsTopic Modeling · AI in Service Interactions · Mental Health via Writing
MethodsFocus
