Adaptive Sampling for Heterogeneous Rank Aggregation from Noisy Pairwise Comparisons

TL;DR

This paper introduces an active sampling method for heterogeneous rank aggregation that adaptively selects users to improve ranking accuracy from noisy pairwise comparisons, with proven theoretical guarantees and empirical advantages.

Contribution

It proposes an elimination-based active sampling strategy that adaptively improves ranking accuracy and reduces sample complexity in noisy, heterogeneous comparison settings.

Findings

Algorithm accurately recovers true rankings with high probability.

Sample complexity is lower than non-active strategies.

Empirical results outperform state-of-the-art baselines.

Abstract

In heterogeneous rank aggregation problems, users often exhibit various accuracy levels when comparing pairs of items. Thus a uniform querying strategy over users may not be optimal. To address this issue, we propose an elimination-based active sampling strategy, which estimates the ranking of items via noisy pairwise comparisons from users and improves the users' average accuracy by maintaining an active set of users. We prove that our algorithm can return the true ranking of items with high probability. We also provide a sample complexity bound for the proposed algorithm which is better than that of non-active strategies in the literature. Experiments are provided to show the empirical advantage of the proposed methods over the state-of-the-art baselines.