Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generation

TL;DR

This paper introduces an atomic-level evaluation framework for assessing persona fidelity in large language models, enabling detection of subtle Out-of-Character behaviors in open-ended generation tasks.

Contribution

It presents a novel, fine-grained evaluation method with three metrics that improve detection of persona deviations over traditional single-score approaches.

Findings

Effective detection of subtle persona inconsistencies

Insights into how task structure affects persona adherence

Identification of challenges in maintaining consistent persona expression

Abstract

Ensuring persona fidelity in large language models (LLMs) is essential for maintaining coherent and engaging human-AI interactions. However, LLMs often exhibit Out-of-Character (OOC) behavior, where generated responses deviate from an assigned persona, leading to inconsistencies that affect model reliability. Existing evaluation methods typically assign single scores to entire responses, struggling to capture subtle persona misalignment, particularly in long-form text generation. To address this limitation, we propose an atomic-level evaluation framework that quantifies persona fidelity at a finer granularity. Our three key metrics measure the degree of persona alignment and consistency within and across generations. Our approach enables a more precise and realistic assessment of persona fidelity by identifying subtle deviations that real users would encounter. Through our experiments, we demonstrate that our framework effectively detects persona inconsistencies that prior methods overlook. By analyzing persona fidelity across diverse tasks and personality types, we reveal how task structure and persona desirability influence model adaptability, highlighting challenges in maintaining consistent persona expression.