Validating Synthetic Usage Data in Living Lab Environments

TL;DR

This paper presents a method to validate synthetic user interaction data generated by click models in data-sparse living lab environments, enabling reliable evaluation of retrieval systems with limited user data.

Contribution

It introduces an evaluation approach for validating click model-generated data against known system rankings in human-in-the-loop settings with sparse data.

Findings

Simple click models can reliably evaluate system performance with 20 sessions.

Complex click models need more data but perform better in simulated experiments.

Distinguishing between diverse systems is easier than reproducing identical rankings.

Abstract

Evaluating retrieval performance without editorial relevance judgments is challenging, but instead, user interactions can be used as relevance signals. Living labs offer a way for small-scale platforms to validate information retrieval systems with real users. If enough user interaction data are available, click models can be parameterized from historical sessions to evaluate systems before exposing users to experimental rankings. However, interaction data are sparse in living labs, and little is studied about how click models can be validated for reliable user simulations when click data are available in moderate amounts. This work introduces an evaluation approach for validating synthetic usage data generated by click models in data-sparse human-in-the-loop environments like living labs. We ground our methodology on the click model's estimates about a system ranking compared to a reference ranking for which the relative performance is known. Our experiments compare different click models and their reliability and robustness as more session log data becomes available. In our setup, simple click models can reliably determine the relative system performance with already 20 logged sessions for 50 queries. In contrast, more complex click models require more session data for reliable estimates, but they are a better choice in simulated interleaving experiments when enough session data are available. While it is easier for click models to distinguish between more diverse systems, it is harder to reproduce the system ranking based on the same retrieval algorithm with different interpolation weights. Our setup is entirely open, and we share the code to reproduce the experiments.