IIRSim Studio: A Dashboard for User Simulation

TL;DR

IIRSim Studio is a web-based dashboard that simplifies user simulation in IR by integrating experiment design, execution, and sharing into a unified, reproducible workflow.

Contribution

It introduces a visual environment, component lifecycle management, provenance modeling, and shared-task workflows for user simulation in IR.

Findings

Supports both novices and experts in simulation design.

Enables sharing and versioning of custom simulation components.

Demonstrated through re-deployment of a micro-task.

Abstract

User simulation is a valuable methodology for evaluation in Information Retrieval (IR), enabling low-cost experimentation and counterfactual analysis. However, existing simulation frameworks are primarily code-centric libraries that require substantial setup effort, which limits adoption and hinders reproducibility. The bottleneck is not the simulation engines themselves, but the lack of infrastructure connecting experiment design, execution, and sharing into a single verifiable workflow. This paper introduces IIRSim Studio, a web-based workbench that addresses this gap through four contributions: (1) a visual environment for composing simulation pipelines on top of simulation frameworks, serving both novices learning simulation concepts and experts piloting large-scale experiments; (2) a component lifecycle that supports authoring, versioning, and sharing custom simulation components through Git-backed storage and runtime injection; (3) a provenance model based on experiment bundles and environment templates that makes the scope of replication explicit; and (4) a shared-task workflow, demonstrated through the re-deployment of a Sim4IA micro-task. IIRSim Studio is available as a hosted service and as a portable containerized deployment.