Getting to "rate-optimal'' in ranking & selection

TL;DR

This paper compares four data-driven ranking and selection policies, analyzing their convergence to theoretical optimal allocations and their empirical performance in selecting the best among multiple simulated systems.

Contribution

It introduces a new gCEI policy and empirically evaluates four policies, highlighting their differences and convergence properties in ranking and selection tasks.

Findings

gCEI converges to the Glynn-Juneja allocation

AOMAP converges to the OCBA optimal allocation

Policies exhibit different behaviors in various settings

Abstract

In their 2004 seminal paper, Glynn and Juneja formally and precisely established the rate-optimal, probability-of-incorrect-selection, replication allocation scheme for selecting the best of k simulated systems. In the case of independent, normally distributed outputs this allocation has a simple form that depends in an intuitively appealing way on the true means and variances. Of course the means and (typically) variances are unknown, but the rate-optimal allocation provides a target for implementable, dynamic, data-driven policies to achieve. In this paper we compare the empirical behavior of four related replication-allocation policies: mCEI from Chen and Rzyhov and our new gCEI policy that both converge to the Glynn and Juneja allocation; AOMAP from Peng and Fu that converges to the OCBA optimal allocation; and TTTS from Russo that targets the rate of convergence of the posterior probability of incorrect selection. We find that these policies have distinctly different behavior in some settings.