Adaptive Sampling for Coarse Ranking

TL;DR

This paper introduces LUCBRank, an efficient algorithm for active coarse ranking that adaptively samples to cluster items by their means, reducing the number of ratings needed in social science applications.

Contribution

The paper presents LUCBRank, a novel PAC algorithm for coarse ranking with theoretical sample complexity bounds and superior empirical performance.

Findings

LUCBRank outperforms existing methods on synthetic data.

It achieves near-optimal sample complexity bounds.

It is effective even without parametric model assumptions.

Abstract

We consider the problem of active coarse ranking, where the goal is to sort items according to their means into clusters of pre-specified sizes, by adaptively sampling from their reward distributions. This setting is useful in many social science applications involving human raters and the approximate rank of every item is desired. Approximate or coarse ranking can significantly reduce the number of ratings required in comparison to the number needed to find an exact ranking. We propose a computationally efficient PAC algorithm LUCBRank for coarse ranking, and derive an upper bound on its sample complexity. We also derive a nearly matching distribution-dependent lower bound. Experiments on synthetic as well as real-world data show that LUCBRank performs better than state-of-the-art baseline methods, even when these methods have the advantage of knowing the underlying parametric model.