Consistency of Ranking Estimators

TL;DR

This paper establishes a general framework for the consistency of empirical Bayesian ranking methods, demonstrating that under certain conditions, these methods reliably rank units as data size grows.

Contribution

It develops a comprehensive framework for the consistency of empirical Bayesian ranking methods, including cases with misspecified priors and loss functions.

Findings

Consistency holds if the loss function is reasonable.

Prior distribution should not be too light-tailed.

Error in measurements must decrease sufficiently fast.

Abstract

The ranking problem is to order a collection of units by some unobserved parameter, based on observations from the associated distribution. This problem arises naturally in a number of contexts, such as business, where we may want to rank potential projects by profitability; or science, where we may want to rank predictors potentially associated with some trait by the strength of the association. This approach provides a valuable alternative to the sparsity framework often used with big data. Most approaches to this problem are empirical Bayesian, where we use the data to estimate the hyperparameters of the prior distribution, then use that distribution to estimate the unobserved parameter values. There are a number of different approaches to this problem, based on different loss functions for mis-ranking units. Despite the number of papers developing methods for this problem, there is no work on the consistency of these methods. In this paper, we develop a general framework for consistency of empirical Bayesian ranking methods, which includes nearly all commonly used methods. We then determine conditions under which consistency holds. Given that little work has been done on selection of prior distribution, and that the loss functions developed are not strongly motivated, we consider the case where both of these are misspecified. We show that provided the loss function is reasonable; the prior distribution is not too light-tailed; and the error in measuring each unit converges to zero at a fast enough rate compared with the number of units (which is assumed to increase to infinity); all ranking methods are consistent.