Persona is a Double-edged Sword: Mitigating the Negative Impact of Role-playing Prompts in Zero-shot Reasoning Tasks

TL;DR

This paper investigates the negative effects of role-playing prompts on LLM reasoning and introduces Jekyll & Hyde, a framework that ensembles role-playing and neutral prompts to improve robustness and reasoning accuracy.

Contribution

The paper presents a novel ensemble framework, Jekyll & Hyde, which predicts appropriate personas and selects better solutions to mitigate negative impacts of role-playing prompts in LLM reasoning.

Findings

Role-playing prompts can degrade reasoning in 7 out of 12 datasets.

Jekyll & Hyde improves reasoning by selecting better solutions from role-playing and neutral prompts.

LLM-generated personas lead to more stable reasoning results than handcrafted personas.

Abstract

Recent studies demonstrate that prompting a role-playing persona to an LLM improves reasoning capability. However, assigning an adequate persona is difficult since LLMs are extremely sensitive to assigned prompts; thus, inaccurately defined personas sometimes hinder LLMs and degrade their reasoning capabilities. In this paper, we first investigate the potential negative impact of injecting persona into language models. Furthermore, we propose a novel framework, Jekyll \& Hyde, which ensembles the outcomes of both role-playing and neutral prompts to enhance the robustness of reasoning ability. Specifically, Jekyll \& Hyde predicts an appropriate persona using an LLM when defining the role-playing prompt. Then, Jekyll \& Hyde collects two potential solutions from role-playing and neutral prompts and selects a better solution using the LLM evaluator. The experimental analysis demonstrates that role-playing prompts sometimes distract LLMs, degrading their reasoning abilities in 7 out of 12 datasets in llama3. Meanwhile, Jekyll \& Hyde improve reasoning capabilities by selecting better choices among the potential solutions on twelve widely-used natural language reasoning datasets. In addition, we reveal that assigning LLM-generated personas obtains more stable results than handcrafted personas.