Persona is a Double-edged Sword: Mitigating the Negative Impact of Role-playing Prompts in Zero-shot Reasoning Tasks
Junseok Kim, Nakyeong Yang, and Kyomin Jung

TL;DR
This paper investigates the negative effects of role-playing prompts on LLM reasoning and introduces Jekyll & Hyde, a framework that ensembles role-playing and neutral prompts to improve robustness and reasoning accuracy.
Contribution
The paper presents a novel ensemble framework, Jekyll & Hyde, which predicts appropriate personas and selects better solutions to mitigate negative impacts of role-playing prompts in LLM reasoning.
Findings
Role-playing prompts can degrade reasoning in 7 out of 12 datasets.
Jekyll & Hyde improves reasoning by selecting better solutions from role-playing and neutral prompts.
LLM-generated personas lead to more stable reasoning results than handcrafted personas.
Abstract
Recent studies demonstrate that prompting a role-playing persona to an LLM improves reasoning capability. However, assigning an adequate persona is difficult since LLMs are extremely sensitive to assigned prompts; thus, inaccurately defined personas sometimes hinder LLMs and degrade their reasoning capabilities. In this paper, we first investigate the potential negative impact of injecting persona into language models. Furthermore, we propose a novel framework, Jekyll \& Hyde, which ensembles the outcomes of both role-playing and neutral prompts to enhance the robustness of reasoning ability. Specifically, Jekyll \& Hyde predicts an appropriate persona using an LLM when defining the role-playing prompt. Then, Jekyll \& Hyde collects two potential solutions from role-playing and neutral prompts and selects a better solution using the LLM evaluator. The experimental analysis demonstrates…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsPersona Design and Applications · Safety Warnings and Signage · Innovative Human-Technology Interaction
MethodsLinear Layer · Residual Connection · Layer Normalization · Multi-Head Attention · Position-Wise Feed-Forward Layer · Adam · Attention Is All You Need · Byte Pair Encoding · Absolute Position Encodings · Softmax
