Replication or exploration? Sequential design for stochastic simulation experiments

TL;DR

This paper develops a sequential design method for noisy computer experiments that balances replication and exploration, improving the accuracy of Gaussian process emulators by adaptively choosing when to replicate or explore new inputs.

Contribution

It introduces a lookahead-based sequential design scheme combined with a heteroskedastic Gaussian process model for efficient learning of signal and noise relationships in simulation experiments.

Findings

Replication improves model accuracy and efficiency.

The proposed method effectively balances exploration and replication.

Performance demonstrated on synthetic and real-world simulation data.

Abstract

We investigate the merits of replication, and provide methods for optimal design (including replicates), with the goal of obtaining globally accurate emulation of noisy computer simulation experiments. We first show that replication can be beneficial from both design and computational perspectives, in the context of Gaussian process surrogate modeling. We then develop a lookahead based sequential design scheme that can determine if a new run should be at an existing input location (i.e., replicate) or at a new one (explore). When paired with a newly developed heteroskedastic Gaussian process model, our dynamic design scheme facilitates learning of signal and noise relationships which can vary throughout the input space. We show that it does so efficiently, on both computational and statistical grounds. In addition to illustrative synthetic examples, we demonstrate performance on two challenging real-data simulation experiments, from inventory management and epidemiology.