Reasoning Does Not Necessarily Improve Role-Playing Ability

TL;DR

This study investigates whether reasoning techniques improve the role-playing abilities of large language models, revealing that certain reasoning methods may hinder performance and identifying future research directions.

Contribution

The paper provides a comprehensive evaluation of reasoning techniques on role-playing LLMs, highlighting limitations of current methods and proposing new research avenues.

Findings

Chain-of-Thought may reduce role-playing performance

Reasoning-optimized LLMs are unsuitable for role-playing

Chinese role-playing outperforms English

Abstract

The application of role-playing large language models (LLMs) is rapidly expanding in both academic and commercial domains, driving an increasing demand for high-precision role-playing models. Simultaneously, the rapid advancement of reasoning techniques has continuously pushed the performance boundaries of LLMs. This intersection of practical role-playing demands and evolving reasoning capabilities raises an important research question: "Can reasoning techniques enhance the role-playing capabilities of LLMs?" To address this, we conduct a comprehensive study using 6 role-playing benchmarks, 24 LLMs, and 3 distinct role-playing strategies, comparing the effectiveness of direct zero-shot role-playing, role-playing with Chain-of-Thought (CoT), and role-playing using reasoning-optimized LLMs. Our findings reveal that CoT may reduce role-playing performance, reasoning-optimized LLMs are unsuitable for role-playing, reasoning ability disrupts the role-playing scaling law, large models still lack proficiency in advanced role-playing, and Chinese role-playing performance surpasses English role-playing performance. Furthermore, based on extensive experimental results, we propose two promising future research directions: Role-aware CoT for improving role-playing LLMs and Reinforcement Learning for role-playing LLMs, aiming to enhance the adaptability, consistency, and effectiveness of role-playing LLMs for both research and real-world applications.