Eval4Sim: An Evaluation Framework for Persona Simulation

TL;DR

Eval4Sim is a comprehensive evaluation framework that measures how well persona-based simulated conversations align with human behavior across adherence, consistency, and naturalness, addressing limitations of existing LLM evaluation methods.

Contribution

It introduces a novel, multi-dimensional evaluation framework for persona simulation that is grounded in observable human conversational patterns and extends beyond scalar scoring.

Findings

Eval4Sim effectively distinguishes between insufficient and over-optimized persona encoding.

The framework provides a more transparent and human-aligned assessment of simulated conversations.

Demonstrated on PersonaChat, Eval4Sim is applicable to any speaker-annotated conversational corpus.

Abstract

Large Language Model (LLM) personas with explicit specifications of attributes, background, and behavioural tendencies are increasingly used to simulate human conversations for tasks such as user modeling, social reasoning, and behavioural analysis. Ensuring that persona-grounded simulations faithfully reflect human conversational behaviour is therefore critical. However, current evaluation practices largely rely on LLM-as-a-judge approaches, offering limited grounding in observable human behavior and producing opaque scalar scores. We address this gap by proposing Eval4Sim, an evaluation framework that measures how closely simulated conversations align with human conversational patterns across three complementary dimensions. Adherence captures how effectively persona backgrounds are implicitly encoded in generated utterances, assessed via dense retrieval with speaker-aware representations. Consistency evaluates whether a persona maintains a distinguishable identity across conversations, computed through authorship verification. Naturalness reflects whether conversations exhibit human-like flow rather than overly rigid or optimized structure, quantified through distributions derived from dialogue-focused Natural Language Inference. Unlike absolute or optimization-oriented metrics, Eval4Sim uses a human conversational corpus (i.e., PersonaChat) as a reference baseline and penalizes deviations in both directions, distinguishing insufficient persona encoding from over-optimized, unnatural behaviour. Although demonstrated on PersonaChat, the applicability of Eval4Sim extends to any conversational corpus containing speaker-level annotations.