Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios

TL;DR

This paper introduces a multi-agent framework for realistic and explainable user simulation in interactive scenarios, leveraging persona control and task state tracking to improve the fidelity and diversity of simulated human interactions.

Contribution

The paper presents a novel multi-agent system with specialized agents for persona control and task tracking, enhancing realism and explainability in user simulation for conversational AI testing.

Findings

Complete system outperforms single-LLM baselines in simulation quality

Significant improvements in persona adherence and task accuracy

Framework demonstrates high realism and behavioral diversity

Abstract

Testing conversational AI systems at scale across diverse domains necessitates realistic and diverse user interactions capturing a wide array of behavioral patterns. We present a novel multi-agent framework for realistic, explainable human user simulation in interactive scenarios, using persona control and task state tracking to mirror human cognitive processes during goal-oriented conversations. Our system employs three specialized AI agents: (1) a User Agent to orchestrate the overall interaction, (2) a State Tracking Agent to maintain structured task state, and (3) a Message Attributes Generation Agent that controls conversational attributes based on task progress and assigned persona. To validate our approach, we implement and evaluate the framework for guest ordering at a restaurant with scenarios rich in task complexity, behavioral diversity, and conversational ambiguity. Through systematic ablations, we evaluate the contributory efficacy of each agentic component to overall simulation quality in terms of persona adherence, task completion accuracy, explainability, and realism. Our experiments demonstrate that the complete multi-agent system achieves superior simulation quality compared to single-LLM baselines, with significant gains across all evaluation metrics. This framework establishes a powerful environment for orchestrating agents to simulate human users with cognitive plausibility, decomposing the simulation into specialized sub-agents that reflect distinct aspects of human thought processes applicable across interactive domains.