Prediction Intervals for Simulation Metamodeling

TL;DR

This paper introduces a neural network-based approach for constructing prediction intervals in simulation metamodeling, offering a scalable, distribution-free method with finite-sample guarantees, and compares it to stochastic kriging.

Contribution

It proposes a novel neural network framework for prediction intervals in metamodeling, addressing scalability and assumption limitations of stochastic kriging.

Findings

Neural network-based intervals achieve accurate coverage and narrow width.

The methods provide distribution-free finite-sample guarantees.

Compared approaches outperform stochastic kriging in numerical tests.

Abstract

Simulation metamodeling refers to the construction of lower-fidelity models to represent input-output relations using few simulation runs. Stochastic kriging, which is based on Gaussian process, is a versatile and common technique for such a task. However, this approach relies on specific model assumptions and could encounter scalability challenges. In this paper, we study an alternative metamodeling approach using prediction intervals to capture the uncertainty of simulation outputs. We cast the metamodeling task as an empirical constrained optimization framework to train prediction intervals that attain accurate prediction coverage and narrow width. We specifically use neural network to represent these intervals and discuss procedures to approximately solve this optimization problem. We also present an adaptation of conformal prediction tools as another approach to construct prediction intervals for metamodeling. Lastly, we present a validation machinery and show how our method can enjoy a distribution-free finite-sample guarantee on the prediction performance. We demonstrate and compare our proposed approaches with existing methods including stochastic kriging through numerical examples.