Prior Distributions for Ranking Problems

TL;DR

This paper investigates how the choice of prior distribution influences Bayesian ranking methods, emphasizing the importance of prior selection beyond conventional conjugate priors in various ranking applications.

Contribution

It provides a detailed analysis of the impact of prior distribution choices on Bayesian ranking performance, offering insights beyond standard conjugate prior approaches.

Findings

Prior choice significantly affects ranking accuracy.

Posterior mean ranking is sensitive to prior distribution.

Guidelines for selecting priors in ranking problems.

Abstract

The ranking problem is to order a collection of units by some unobserved parameter, based on observations from the associated distribution. This problem arises naturally in a number of contexts, such as business, where we may want to rank potential projects by profitability; or science, where we may want to rank variables potentially associated with some trait by the strength of the association. Most approaches to this problem are empirical Bayesian, where we use the data to estimate the hyperparameters of the prior distribution, then use that distribution to estimate the unobserved parameter values. There are a number of different approaches to this problem, based on different loss functions for mis-ranking units. However, little has been done on the choice of prior distribution. Typical approaches involve choosing a conjugate prior for convenience, and estimating the hyperparameters by MLE from the whole dataset. In this paper, we look in more detail at the effect of choice of prior distribution on Bayesian ranking. We focus on the use of posterior mean for ranking, but many of our conclusions should apply to other ranking criteria, and it is not too difficult to adapt our methods to other choices of prior distributions.