Bayesian ranking and selection with applications to field studies, economic mobility, and forecasting

TL;DR

This paper introduces Bayesian algorithms for ranking and selecting candidates based on noisy estimates, demonstrating improved performance over traditional methods in various real-world applications and providing a user-friendly Python package.

Contribution

The paper develops novel Bayesian ranking and selection algorithms that outperform frequentist methods and offers a practical implementation in Python.

Findings

Bayesian algorithms produce shorter confidence intervals.

Bayesian methods select more candidates with correct error rates.

Algorithms outperform frequentist approaches in simulations.

Abstract

Decision-making often involves ranking and selection. For example, to assemble a team of political forecasters, we might begin by narrowing our choice set to the candidates we are confident rank among the top 10% in forecasting ability. Unfortunately, we do not know each candidate's true ability but observe a noisy estimate of it. This paper develops new Bayesian algorithms to rank and select candidates based on noisy estimates. Using simulations based on empirical data, we show that our algorithms often outperform frequentist ranking and selection algorithms. Our Bayesian ranking algorithms yield shorter rank confidence intervals while maintaining approximately correct coverage. Our Bayesian selection algorithms select more candidates while maintaining correct error rates. We apply our ranking and selection procedures to field experiments, economic mobility, forecasting, and similar problems. Finally, we implement our ranking and selection techniques in a user-friendly Python package documented here: https://dsbowen-conditional-inference.readthedocs.io/en/latest/.