Evaluation of Temporal Change in IR Test Collections

TL;DR

This paper explores how temporal changes in document collections impact the evaluation of information retrieval systems, proposing a model to assess effectiveness over time and demonstrating significant effects of data changes on retrieval performance.

Contribution

It introduces a conceptual model extending Cranfield experiments to include temporal dynamics and evaluates how different types of data changes affect retrieval effectiveness.

Findings

Retrieval effectiveness varies significantly with different data change scenarios.

Both average and relative system performances are affected by the extent and type of data changes.

Proposed measures effectively describe changes in retrieval results over time.

Abstract

Information retrieval systems have been evaluated using the Cranfield paradigm for many years. This paradigm allows a systematic, fair, and reproducible evaluation of different retrieval methods in fixed experimental environments. However, real-world retrieval systems must cope with dynamic environments and temporal changes that affect the document collection, topical trends, and the individual user's perception of what is considered relevant. Yet, the temporal dimension in IR evaluations is still understudied. To this end, this work investigates how the temporal generalizability of effectiveness evaluations can be assessed. As a conceptual model, we generalize Cranfield-type experiments to the temporal context by classifying the change in the essential components according to the create, update, and delete operations of persistent storage known from CRUD. From the different types of change different evaluation scenarios are derived and it is outlined what they imply. Based on these scenarios, renowned state-of-the-art retrieval systems are tested and it is investigated how the retrieval effectiveness changes on different levels of granularity. We show that the proposed measures can be well adapted to describe the changes in the retrieval results. The experiments conducted confirm that the retrieval effectiveness strongly depends on the evaluation scenario investigated. We find that not only the average retrieval performance of single systems but also the relative system performance are strongly affected by the components that change and to what extent these components changed.