Using Context to Improve the Evaluation of Information Retrieval Systems

TL;DR

This paper proposes a novel context-aware evaluation approach for information retrieval systems, addressing limitations of traditional methods by considering user context, query-document adequacy, and search tool characteristics, demonstrated through experiments with major search engines.

Contribution

It introduces a new multi-level evaluation method incorporating context, improving the assessment of search engine performance beyond classic approaches.

Findings

Search engines' ability to reject irrelevant results depends on query formulation and site policies.

Context-aware evaluation reveals a decline in perceived relevance as more results are considered.

The approach is applicable to real-world search tools like Google, Bing, and Yahoo.

Abstract

The crucial role of the evaluation in the development of the information retrieval tools is useful evidence to improve the performance of these tools and the quality of results that they return. However, the classic evaluation approaches have limitations and shortcomings especially regarding to the user consideration, the measure of the adequacy between the query and the returned documents and the consideration of characteristics, specifications and behaviors of the search tool. Therefore, we believe that the exploitation of contextual elements could be a very good way to evaluate the search tools. So, this paper presents a new approach that takes into account the context during the evaluation process at three complementary levels. The experiments gives at the end of this article has shown the applicability of the proposed approach to real research tools. The tests were performed with the most popular searching engine (i.e. Google, Bing and Yahoo) selected in particular for their high selectivity. The obtained results revealed that the ability of these engines to rejecting dead links, redundant results and parasites pages depends strongly to how queries are formulated, and to the political of sites offering this information to present their content. The relevance evaluation of results provided by these engines, using the user's judgments, then using an automatic manner to take into account the query context has also shown a general decline in the perceived relevance according to the number of the considered results.