A Novelty-based Evaluation Method for Information Retrieval

TL;DR

This paper introduces a new evaluation method for information retrieval systems that emphasizes retrieving novel relevant documents, addressing the limitations of traditional metrics like precision and recall.

Contribution

The paper proposes a novel evaluation approach that rewards systems for retrieving unique, previously unretrieved relevant documents, enhancing the assessment of IR system novelty.

Findings

The method effectively distinguishes systems based on their ability to retrieve novel documents.

Evaluation results show improved identification of innovative IR systems.

The approach was validated using systems from the IREX workshop.

Abstract

In information retrieval research, precision and recall have long been used to evaluate IR systems. However, given that a number of retrieval systems resembling one another are already available to the public, it is valuable to retrieve novel relevant documents, i.e., documents that cannot be retrieved by those existing systems. In view of this problem, we propose an evaluation method that favors systems retrieving as many novel documents as possible. We also used our method to evaluate systems that participated in the IREX workshop.