Investigating Order Effects in Multidimensional Relevance Judgment using Query Logs

TL;DR

This paper introduces a methodology to analyze how the order of relevance judgments affects user decisions in web search, using large-scale query logs and quantum cognitive models to detect irrational behaviors.

Contribution

It presents a scalable approach to study order effects in relevance judgment with real-world data and offers a quantum cognitive framework for explanation.

Findings

Detected order effects in relevance judgments from query logs

Validated irrational user behavior in relevance assessment

Applied quantum cognitive models to explain observed effects

Abstract

There is a growing body of research which has investigated relevance judgment in IR being influenced by multiple factors or dimensions. At the same time, the Order Effects in sequential decision making have been quantitatively detected and studied in Mathematical Psychology. Combining the two phenomena, there have been some user studies carried out which investigate the Order Effects and thus incompatibility in different dimensions of relevance. In this work, we propose a methodology for carrying out such an investigation in large scale and real world data using query logs of a web search engine, and device a test to detect the presence of an irrational user behavior in relevance judgment of documents. We further validate this behavior through a Quantum Cognitive explanation of the Order and Context effects.