Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMs

TL;DR

This paper introduces a standardized framework to analyze how consistently large language models adhere to assigned personas across various tasks and categories, revealing factors that influence persona consistency.

Contribution

The paper presents a new framework for evaluating persona consistency in LLMs across multiple categories and task types, addressing a gap in comprehensive analysis.

Findings

Consistency varies with persona stereotypes and model design.

Structured tasks and more context improve consistency.

Model adherence to personas is influenced by multiple factors.

Abstract

Personalized Large Language Models (LLMs) are increasingly used in diverse applications, where they are assigned a specific persona - such as a happy high school teacher - to guide their responses. While prior research has examined how well LLMs adhere to predefined personas in writing style, a comprehensive analysis of consistency across different personas and task types is lacking. In this paper, we introduce a new standardized framework to analyze consistency in persona-assigned LLMs. We define consistency as the extent to which a model maintains coherent responses when assigned the same persona across different tasks and runs. Our framework evaluates personas across four different categories (happiness, occupation, personality, and political stance) spanning multiple task dimensions (survey writing, essay generation, social media post generation, single turn, and multi-turn conversations). Our findings reveal that consistency is influenced by multiple factors, including the assigned persona, stereotypes, and model design choices. Consistency also varies across tasks, increasing with more structured tasks and additional context. All code is available on GitHub.