Uncertainty in Ranking

TL;DR

This paper characterizes the uncertainty in estimated ranks derived from data, providing methods to quantify and construct confidence sets for rankings using interval estimates and efficient algorithms.

Contribution

It introduces a complete characterization of rank uncertainty based on linear extensions of partial orders and develops a set estimator with confidence guarantees.

Findings

The set estimator is a valid confidence set for rankings.

Efficient algorithms are provided for key ranking questions.

The methods are demonstrated on simulated and real data.

Abstract

Ranks estimated from data are uncertain and this poses a challenge in many applications. However, estimated ranks are deterministic functions of estimated parameters, so the uncertainty in the ranks must be determined by the uncertainty in the parameter estimates. We give a complete characterization of this relationship in terms of the linear extensions of a partial order determined by interval estimates of the parameters of interest. We then use this relationship to give a set estimator for the overall ranking, use its size to measure the uncertainty in a ranking, and give efficient algorithms for several questions of interest. We show that our set estimator is a valid confidence set and describe its relationship to a joint confidence set for ranks recently proposed by Klein, Wright \& Wieczorek. We apply our methods to both simulated and real data and make them available through the R package rankUncertainty.