Exploiting Hierarchical Dependence Structures for Unsupervised Rank Fusion in Information Retrieval

TL;DR

This paper introduces an unsupervised, nonlinear rank fusion method based on copula theory that models complex dependencies between retrieval results, improving IR performance in specific scenarios.

Contribution

It presents a novel copula-inspired, dynamic, nonlinear rank fusion approach that accounts for query-specific dependencies in information retrieval.

Findings

Improves IR performance over traditional methods in certain conditions.

Models complex, nonlinear dependencies between search results.

Outperforms CombMNZ and other nonlinear strategies in experiments.

Abstract

The goal of rank fusion in information retrieval (IR) is to deliver a single output list from multiple search results. Improving performance by combining the outputs of various IR systems is a challenging task. A central point is the fact that many non-obvious factors are involved in the estimation of relevance, inducing nonlinear interrelations between the data. The ability to model complex dependency relationships between random variables has become increasingly popular in the realm of information retrieval, and the need to further explore these dependencies for data fusion has been recently acknowledged. Copulas provide a framework to separate the dependence structure from the margins. Inspired by the theory of copulas, we propose a new unsupervised, dynamic, nonlinear, rank fusion method, based on a nested composition of non-algebraic function pairs. The dependence structure of the model is tailored by leveraging query-document correlations on a per-query basis. We experimented with three topic sets over CLEF corpora fusing 3 and 6 retrieval systems, comparing our method against the CombMNZ technique and other nonlinear unsupervised strategies. The experiments show that our fusion approach improves performance under explicit conditions, providing insight about the circumstances under which linear fusion techniques have comparable performance to nonlinear methods.