Statistical Uncertainty Analysis for Stochastic Simulation

TL;DR

This paper introduces a method to accurately quantify total statistical uncertainty in stochastic simulation outputs by combining bootstrapping, stochastic kriging metamodeling, and variance decomposition, addressing both simulation and input data uncertainties.

Contribution

It presents a novel approach that integrates bootstrapping and stochastic kriging to produce confidence intervals accounting for all sources of uncertainty in simulation performance estimates.

Findings

The method effectively captures combined simulation and input uncertainty.

Variance decomposition identifies the dominant source of uncertainty.

Empirical results show good finite-sample performance of the approach.

Abstract

When we use simulation to evaluate the performance of a stochastic system, the simulation often contains input distributions estimated from real-world data; therefore, there is both simulation and input uncertainty in the performance estimates. Ignoring either source of uncertainty underestimates the overall statistical error. Simulation uncertainty can be reduced by additional computation (e.g., more replications). Input uncertainty can be reduced by collecting more real-world data, when feasible. This paper proposes an approach to quantify overall statistical uncertainty when the simulation is driven by independent parametric input distributions; specifically, we produce a confidence interval that accounts for both simulation and input uncertainty by using a metamodel-assisted bootstrapping approach. The input uncertainty is measured via bootstrapping, an equation-based stochastic kriging metamodel propagates the input uncertainty to the output mean, and both simulation and metamodel uncertainty are derived using properties of the metamodel. A variance decomposition is proposed to estimate the relative contribution of input to overall uncertainty; this information indicates whether the overall uncertainty can be significantly reduced through additional simulation alone. Asymptotic analysis provides theoretical support for our approach, while an empirical study demonstrates that it has good finite-sample performance.