RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems

TL;DR

This paper introduces RoleRMBench, a benchmark for reward modeling in role-playing dialogue, and proposes RoleRM, a new reward model trained with continuous implicit preferences, significantly improving alignment with human judgments.

Contribution

The paper presents the first benchmark for reward modeling in role play and introduces RoleRM, a novel reward model utilizing continuous implicit preferences for better subjective alignment.

Findings

RoleRM outperforms existing reward models by over 24% on average.

Large gaps exist between general reward models and human judgments in role play.

Continuous implicit preferences improve subjective evaluation consistency.

Abstract

Reward modeling has become a cornerstone of aligning large language models (LLMs) with human preferences. Yet, when extended to subjective and open-ended domains such as role play, existing reward models exhibit severe degradation, struggling to capture nuanced and persona-grounded human judgments. To address this gap, we introduce RoleRMBench, the first systematic benchmark for reward modeling in role-playing dialogue, covering seven fine-grained capabilities from narrative management to role consistency and engagement. Evaluation on RoleRMBench reveals large and consistent gaps between general-purpose reward models and human judgment, particularly in narrative and stylistic dimensions. We further propose RoleRM, a reward model trained with Continuous Implicit Preferences (CIP), which reformulates subjective evaluation as continuous consistent pairwise supervision under multiple structuring strategies. Comprehensive experiments show that RoleRM surpasses strong open- and closed-source reward models by over 24% on average, demonstrating substantial gains in narrative coherence and stylistic fidelity. Our findings highlight the importance of continuous preference representation and annotation consistency, establishing a foundation for subjective alignment in human-centered dialogue systems.