Optimizing Input Data Collection for Ranking and Selection

TL;DR

This paper develops an optimal sequential sampling algorithm for ranking and selection problems with Bayesian input models, improving efficiency and extending to continuous input spaces with strong consistency guarantees.

Contribution

It introduces OSAR, an optimal sampling algorithm that achieves near-optimal input and simulation data ratios, and extends it with kernel ridge regression for continuous input spaces.

Findings

OSAR achieves epsilon-optimal ratios almost surely.

Kernel ridge regression enhances finite-sample performance.

OSAR outperforms state-of-the-art methods in numerical tests.

Abstract

We study a ranking and selection (R&S) problem when all solutions share common parametric Bayesian input models updated with the data collected from multiple independent data-generating sources. Our objective is to identify the best system by designing a sequential sampling algorithm that collects input and simulation data given a budget. We adopt the most probable best (MPB) as the estimator of the optimum and show that its posterior probability of optimality converges to one at an exponential rate as the sampling budget increases. Assuming that the input parameters belong to a finite set, we characterize the $\epsilon$-optimal static sampling ratios for input and simulation data that maximize the convergence rate. Using these ratios as guidance, we propose the optimal sampling algorithm for R&S (OSAR) that achieves the $\epsilon$-optimal ratios almost surely in the limit. We further extend OSAR by adopting the kernel ridge regression to improve the simulation output mean prediction. This not only improves OSAR's finite-sample performance, but also lets us tackle the case where the input parameters lie in a continuous space with a strong consistency guarantee for finding the optimum. We numerically demonstrate that OSAR outperforms a state-of-the-art competitor.