Role-Playing Evaluation for Large Language Models

TL;DR

This paper introduces RPEval, a new benchmark for evaluating large language models' role-playing abilities across emotional, decision-making, moral, and consistency dimensions, addressing evaluation challenges.

Contribution

The paper presents RPEval, a comprehensive benchmark for assessing LLM role-playing skills, including its construction and baseline evaluations.

Findings

RPEval effectively measures LLM role-playing capabilities.

Baseline results highlight strengths and weaknesses of current models.

The benchmark is publicly available for further research.

Abstract

Large Language Models (LLMs) demonstrate a notable capacity for adopting personas and engaging in role-playing. However, evaluating this ability presents significant challenges, as human assessments are resource-intensive and automated evaluations can be biased. To address this, we introduce Role-Playing Eval (RPEval), a novel benchmark designed to assess LLM role-playing capabilities across four key dimensions: emotional understanding, decision-making, moral alignment, and in-character consistency. This article details the construction of RPEval and presents baseline evaluations. Our code and dataset are available at https://github.com/yelboudouri/RPEval