A Usefulness-based Approach for Measuring the Local and Global Effect of IIR Services

TL;DR

This paper introduces a usefulness-based evaluation method for IIR services that assesses their impact on the entire search process using log data, providing a more comprehensive measure than traditional relevance or usability tests.

Contribution

It operationalizes the usefulness model at the system support level and applies log-based analysis to measure local and global effects of IIR services.

Findings

Search term suggestion significantly increases positive signals in subsequent search steps.

The log-based approach effectively captures the impact of IIR services on user search behavior.

The method offers a scalable alternative to traditional evaluation techniques.

Abstract

In Interactive Information Retrieval (IIR) different services such as search term suggestion can support users in their search process. The applicability and performance of such services is either measured with different user-centered studies (like usability tests or laboratory experiments) or, in the context of IR, with their contribution to measures like precision and recall. However, each evaluation methodology has its certain disadvantages. For example, user-centered experiments are often costly and small-scaled; IR experiments rely on relevance assessments and measure only relevance of documents. In this work we operationalize the usefulness model of Cole et al. (2009) on the level of system support to measure not only the local effect of an IR service, but the impact it has on the whole search process. We therefore use a log-based evaluation approach which models user interactions within sessions with positive signals and apply it for the case of a search term suggestion service. We found that the usage of the service significantly often implicates the occurrence of positive signals during the following session steps.