A Framework for Evaluating the Retrieval Effectiveness of Search Engines

TL;DR

This paper introduces a theoretical framework for assessing the effectiveness of next-generation search engines, especially those with enriched result presentations, applicable across various search domains.

Contribution

It proposes a comprehensive framework that evaluates both presentation and retrieval effectiveness, extending traditional testing methods for modern, enriched search interfaces.

Findings

Framework applicable to diverse search engines

Addresses influence of presentation on retrieval effectiveness

Guidelines for designing reliable evaluation tests

Abstract

This chapter presents a theoretical framework for evaluating next generation search engines. We focus on search engines whose results presentation is enriched with additional information and does not merely present the usual list of 10 blue links, that is, of ten links to results, accompanied by a short description. While Web search is used as an example here, the framework can easily be applied to search engines in any other area. The framework not only addresses the results presentation, but also takes into account an extension of the general design of retrieval effectiveness tests. The chapter examines the ways in which this design might influence the results of such studies and how a reliable test is best designed.