Data-driven Ranking and Selection under Input Uncertainty

TL;DR

This paper develops new simulation-based ranking and selection methods that adaptively handle input uncertainty and batch data, enabling efficient identification of the best design with fewer simulations.

Contribution

It introduces a moving average estimator and sequential elimination procedures with confidence bands that explicitly account for input uncertainty in an online setting.

Findings

The proposed methods effectively identify the best design with fewer batches.

Incorporating input uncertainty improves decision confidence.

Optimizing the drop rate enhances simulation efficiency.

Abstract

We consider a simulation-based Ranking and Selection (R&S) problem with input uncertainty, where unknown input distributions can be estimated using input data arriving in batches of varying sizes over time. Each time a batch arrives, additional simulations can be run using updated input distribution estimates. The goal is to confidently identify the best design after collecting as few batches as possible. We first introduce a moving average estimator for aggregating simulation outputs generated under heterogenous input distributions. Then, based on a Sequential Elimination framework, we devise two major R&S procedures by establishing exact and asymptotic confidence bands for the estimator. In deriving the latter confidence bands, we incorporate the result of "Multiple Comparison with Best" and establish an asymptotic normality result which explicitly characterizes the tradeoff between input uncertainty and stochastic uncertainty in an online environment. We also extend our procedures to the indifference zone setting, which helps save simulation effort for practical usage. Numerical results show the effectiveness and necessity of our procedures. Moreover, the efficiency can be further boosted through optimizing the "drop rate" parameter of the estimator.