Empirical Bayes Selection for Value Maximization

TL;DR

This paper analyzes empirical Bayes methods for selecting the top units to maximize total value, showing they perform near-optimally with regret bounds and confirming effectiveness through real internet experiment data.

Contribution

It establishes regret bounds for empirical Bayes selection and demonstrates their near-optimality, supported by real-world internet experiment data.

Findings

Empirical Bayes selection incurs $O_p(n^{-1})$ regret relative to the Bayesian oracle.

Regret is $O_p(r_n^2)$ when the prior estimate error is $O_p(r_n)$.

Empirical Bayes methods perform well in real internet experiments with modest data.

Abstract

We study the problem of selecting the best $m$ units from a set of $n$ as $m / n \to \alpha \in (0, 1)$, where noisy, heteroskedastic measurements of the units' true values are available and the decision-maker wishes to maximize the aggregate true value of the units selected. Given a parametric prior distribution, the empirical Bayes decision rule incurs $O_p(n^{-1})$ regret relative to the Bayesian oracle that knows the true prior. More generally, if the error in the estimated prior is of order $O_p(r_n)$, regret is $O_p(r_n^2)$. In this sense \emph{selection} of the best units is fundamentally easier than \emph{estimation} of their values. We show this regret bound is sharp in the parametric case, by giving an example in which it is attained. Using priors calibrated from a dataset of over four thousand internet experiments, we confirm that empirical Bayes methods perform well in detecting the best treatments with only a modest number of experiments.