PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models

TL;DR

PersonaArena is a dynamic simulation framework that evaluates and improves persona-level role-playing in large language models through multi-turn interactions and multi-agent assessments.

Contribution

It introduces a novel simulation environment using social content to enhance LLMs' social role-playing capabilities and evaluation methods.

Findings

Enables rigorous assessment of LLMs' role-playing abilities.

Improves LLMs' authenticity and social adeptness.

Provides a multi-agent debating judge for unbiased evaluation.

Abstract

Large language models (LLMs) increasingly serve as interactive social agents, yet their ability to maintain coherent and authentic persona-level role-playing remains limited, particularly in realistic social scenarios. Existing research predominantly focuses on character-level settings and relies on static evaluation formats, failing to capture the complexity of everyday social interactions. In this work, we present PersonaArena, a dynamic simulation framework for evaluating and improving persona-level role-playing in LLMs. PersonaArena leverages a large, filtered corpus of user-generated social content to construct a nuanced persona bank, and elicits multi-turn, context-rich interactions within simulated social environments. Our framework features a multi-agent debating judge for holistic and unbiased assessment. Through extensive experiments, we demonstrate that PersonaArena enables rigorous evaluation and enhancement of LLMs' role-playing capabilities, advancing the development of more authentic and socially adept AI agents.