Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning

TL;DR

This paper presents a reinforcement learning framework to enhance the consistency of human personas in dialogue generated by large language models, improving their coherence and faithfulness across multiple turns.

Contribution

It introduces a unified evaluation framework with new metrics and applies multi-turn reinforcement learning to significantly reduce persona drift in LLMs.

Findings

Persona inconsistency reduced by over 55%.

Enhanced coherence and faithfulness in simulated dialogues.

Validated metrics align well with human judgments.

Abstract

Large Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play. While these simulations enable scalable training and evaluation of AI agents, off-the-shelf LLMs often drift from their assigned personas, contradict earlier statements, or abandon role-appropriate behavior. We introduce a unified framework for evaluating and improving persona consistency in LLM-generated dialogue. We define three automatic metrics: prompt-to-line consistency, line-to-line consistency, and Q&A consistency, that capture different types of persona drift and validate each against human annotations. Using these metrics as reward signals, we apply multi-turn reinforcement learning to fine-tune LLMs for three user roles: a patient, a student, and a social chat partner. Our method reduces inconsistency by over 55%, resulting in more coherent and faithful simulated users.