Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants

TL;DR

This paper investigates how the quality of user simulators for training LLM assistants can be measured by their effectiveness in real-world interactions, emphasizing the importance of real human grounding.

Contribution

It introduces a method to quantify simulator quality based on downstream performance of trained assistants with real users and compares different simulator training approaches.

Findings

Training with fine-tuned simulators improves assistant performance in user studies.

Role-playing LLMs benefit from persona conditioning but do not match fine-tuned simulators.

Scaling simulator size enhances fine-tuned models but not role-playing ones.

Abstract

User simulators are increasingly leveraged to build interactive AI assistants, yet how to measure the quality of these simulators remains an open question. In this work, we show how simulator quality can be quantified in terms of its downstream utility: how an LLM assistant trained with this user simulator performs in the wild when interacting with real humans. In a controlled experiment where only the user simulator varies, we train LLM assistants via reinforcement learning against a spectrum of simulators, from an LLM prompted to role-play a user to one fine-tuned on human utterances from WildChat. As evaluation, we measure pairwise win rates in a user study with 283 participants and on WildBench, a benchmark derived from real human--AI conversations. Training against the role-playing LLM yields an assistant statistically indistinguishable from the initial assistant in our user study (51% win rate), whereas training against the fine-tuned simulator yields significant gains (58% over the initial and 57% over the one trained against role-playing). Closer inspection reveals three further patterns: methods for making role-playing LLMs more realistic (e.g., persona conditioning) improve trained assistants but do not close the gap to the fine-tuned simulator; scaling the simulator's model size benefits the fine-tuned simulator but yields no gain for role-playing ones; and assistants trained against role-playing simulators fail to generalize when paired with other simulators at test time, while the one trained against fine-tuned simulator does. Together, these results argue for grounding user simulators in real human behavior and measuring their quality by their downstream effect on real users.