Do Persona-Infused LLMs Affect Performance in a Strategic Reasoning Game?

TL;DR

This paper examines how persona prompting in large language models influences strategic decision-making in a competitive game, introducing a mediator-based translation method to improve heuristic reliability and understanding.

Contribution

It presents a novel mediator approach inspired by psychometric analysis to translate persona prompts into effective heuristics for strategic performance evaluation.

Findings

Certain personas enhance game performance when mediated

Mediator improves heuristic reliability and face validity

Persona prompting influences decision-making in strategic environments

Abstract

Although persona prompting in large language models appears to trigger different styles of generated text, it is unclear whether these translate into measurable behavioral differences, much less whether they affect decision-making in an adversarial strategic environment that we provide as open-source. We investigate the impact of persona prompting on strategic performance in PERIL, a world-domination board game. Specifically, we compare the effectiveness of persona-derived heuristic strategies to those chosen manually. Our findings reveal that certain personas associated with strategic thinking improve game performance, but only when a mediator is used to translate personas into heuristic values. We introduce this mediator as a structured translation process, inspired by exploratory factor analysis, that maps LLM-generated inventory responses into heuristics. Results indicate our method enhances heuristic reliability and face validity compared to directly inferred heuristics, allowing us to better study the effect of persona types on decision making. These insights advance our understanding of how persona prompting influences LLM-based decision-making and propose a heuristic generation method that applies psychometric principles to LLMs.