Challenges for Measuring Usefulness of Interactive IR Systems with Log-based Approaches

TL;DR

This paper discusses the challenges of evaluating the usefulness of interactive information retrieval systems using log data, aiming for a more human-centered and operationalizable approach beyond traditional user studies.

Contribution

It highlights the difficulties in operationalizing the usefulness model with log data and discusses potential approaches for more practical, comparable evaluations.

Findings

Log-based evaluation faces significant operationalization challenges.

Explicit user studies provide detailed data but are limited outside labs.

Operationalizing the usefulness model could enable broader system comparisons.

Abstract

The usefulness evaluation model proposed by Cole et al. in 2009 [2] focuses on the evaluation of interactive IR systems by their support towards the user's overall goal, sub goals and tasks. This is a more human focus of the IR evaluation process than with classical TREC-oriented studies and gives a more holistic view on the IR evaluation process. However, yet there is no formal framework how the usefulness model can be operationalized. Additionally, a lot of information needed for the operationalization is only available in explicit user studies where for example the overall goal and the tasks are prompted from the users or are predefined. Measuring the usefulness of IR systems outside the laboratory is a challenging task as most often only log data of user interaction is available. But, an operationalization of the usefulness model based on interaction data could be applied to diverse systems and evaluation results would be comparable. In this article we discuss the challenges for measuring the usefulness of IIR systems with log-based approaches.