MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likeness

TL;DR

MirrorBench is a comprehensive benchmarking framework designed to evaluate the human-likeness of conversational user proxy agents using diverse lexical and judge-based metrics, independent of task success.

Contribution

It introduces a reproducible, extensible framework with novel metrics and calibration controls for assessing user proxies' human-likeness in dialogue systems.

Findings

MirrorBench reveals systematic gaps between user proxies and real humans.

The framework provides variance-aware comparisons across datasets.

Open source implementation facilitates reproducibility and further research.

Abstract

Large language models (LLMs) are increasingly used as human simulators, both for evaluating conversational systems and for generating fine-tuning data. However, naive "act-as-a-user" prompting often yields verbose, unrealistic utterances, motivating principled evaluation of *user proxy agents*. We present **MirrorBench**, a reproducible and extensible benchmarking framework that evaluates user proxies solely on their ability to produce human-like user utterances across diverse conversational regimes, explicitly decoupled from downstream task success. **MirrorBench** combines three lexical-diversity metrics (**MATTR**, **Yule's~$K$**, and **HD-D**) with three LLM-judge-based metrics (**GTEval**, **Pairwise Indistinguishability**, and **Rubric-and-Reason**), and contextualizes judge scores using Human-Human and Proxy-Proxy calibration controls. Across four public datasets, **MirrorBench** yields variance-aware comparisons and reveals systematic gaps between user proxies and real human users. The framework is open sourced at https://github.com/SAP/mirrorbench and includes a command-line interface for running and managing user-proxy benchmarking experiments.