SimEval-IR: A Unified Toolkit and Benchmark Suite for Evaluating User Simulators and Search Sessions

TL;DR

SimEval-IR is an open-source toolkit and benchmark suite that standardizes evaluation of user simulators in information retrieval, distinguishing behavioral realism from tester reliability, and providing concrete metrics and baseline results.

Contribution

It introduces a unified session schema, three comprehensive benchmarks, and baseline results, addressing the lack of standardized evaluation tools for user simulators in IR.

Findings

Classifier-discriminator 'human-likeness' check has low predictive power for system ranking.

Distance metrics like click-depth and session embedding Fréchet distance are more effective.

SimEval-IR is publicly available with configurations and scripts for reproducibility.

Abstract

User simulators are increasingly central to interactive information retrieval, yet the community lacks standardized evaluation tools. Simulators serve two objectives, behavioral realism (matching real user behavior) and tester reliability (producing valid system rankings), and these are often conflated despite being distinct and sometimes conflicting. We present SimEval-IR, an open-source toolkit and benchmark suite that makes this distinction measurable. SimEval-IR provides: (1) a canonical session schema unifying session search and conversational interactions, with validated dataset adapters and explicit loss accounting; (2) three executable benchmarks covering behavioral realism, tester reliability with RATE-style estimation, and an analysis linking the two; and (3) baseline results across four real datasets in two languages and four simulator families. Our key finding: the classifier-discriminator ''human-likeness'' check, the dominant realism test in the literature, has essentially no pooled predictive power for system-ranking validity ($r{=}{+}0.09$, $n{=}48$), while marginal click-depth distance and Fr\'{e}chet distance over session embeddings give a much stronger signal ($|r|{=}0.43$ and $0.40$, $p{\leq}0.005$). SimEval-IR is released with all configurations and scripts to reproduce the reported analysis.