A Formal Account of Effectiveness Evaluation and Ranking Fusion

TL;DR

This paper introduces a theoretical information-theoretic framework for evaluating and fusing retrieval system effectiveness, improving upon existing metrics and explaining the benefits of combining system outputs.

Contribution

It formalizes effectiveness and ranking fusion using Information Theory, providing a rigorous foundation and improving metric constraints.

Findings

Proposed effectiveness metric outperforms popular metrics.

Empirical results show the metric captures traditional quality aspects.

The framework explains effectiveness gains from unsupervised output fusion.

Abstract

This paper proposes a theoretical framework which models the information provided by retrieval systems in terms of Information Theory. The proposed framework allows to formalize: (i) system effectiveness as an information theoretic similarity between system outputs and human assessments, and (ii) ranking fusion as an information quantity measure. As a result, the proposed effectiveness metric improves popular metrics in terms of formal constraints. In addition, our empirical experiments suggest that it captures quality aspects from traditional metrics, while the reverse is not true. Our work also advances the understanding of theoretical foundations of the empirically known phenomenon of effectiveness increase when combining retrieval system outputs in an unsupervised manner.