Raising Reliability of Web Search Tool Research through Replication and Chaos Theory

TL;DR

This paper demonstrates that repeated replication of Web search tool evaluations can transform initially unreliable results into consistent and reliable findings, addressing the challenges posed by the Web's dynamic nature.

Contribution

It introduces a replication-based approach to improve the reliability of Web search tool evaluations, applying chaos theory principles to the evaluation process.

Findings

Replication increases result reliability

Multiple iterations lead to consistent outcomes

Web search evaluations become more dependable

Abstract

Because the World Wide Web is a dynamic collection of information, the Web search tools (or "search engines") that index the Web are dynamic. Traditional information retrieval evaluation techniques may not provide reliable results when applied to the Web search tools. This study is the result of ten replications of the classic 1996 Ding and Marchionini Web search tool research. It explores the effects that replication can have on transforming unreliable results from one iteration into replicable and therefore reliable results after multiple iterations.