Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?

TL;DR

This paper evaluates whether large language models can accurately simulate human decision-making based on character personas in literature, introducing a new dataset and a method that improves persona-driven decision accuracy.

Contribution

It introduces the LIFECHOICE dataset for persona-based decision prediction and proposes CHARMAP, a retrieval-based method that enhances decision accuracy in role-playing language agents.

Findings

State-of-the-art LLMs show promising decision-making abilities.

The CHARMAP method improves accuracy by 5.03%.

Significant room for improvement in persona-driven decision modeling.

Abstract

Can Large Language Models (LLMs) simulate humans in making important decisions? Recent research has unveiled the potential of using LLMs to develop role-playing language agents (RPLAs), mimicking mainly the knowledge and tones of various characters. However, imitative decision-making necessitates a more nuanced understanding of personas. In this paper, we benchmark the ability of LLMs in persona-driven decision-making. Specifically, we investigate whether LLMs can predict characters' decisions provided by the preceding stories in high-quality novels. Leveraging character analyses written by literary experts, we construct a dataset LIFECHOICE comprising 1,462 characters' decision points from 388 books. Then, we conduct comprehensive experiments on LIFECHOICE, with various LLMs and RPLA methodologies. The results demonstrate that state-of-the-art LLMs exhibit promising capabilities in this task, yet substantial room for improvement remains. Hence, we further propose the CHARMAP method, which adopts persona-based memory retrieval and significantly advances RPLAs on this task, achieving 5.03% increase in accuracy.