Robust Output Analysis with Monte-Carlo Methodology

TL;DR

This paper introduces a nonparametric Monte Carlo-based framework for output analysis in simulation and machine learning, improving robustness and accuracy of confidence intervals by addressing input uncertainty.

Contribution

It presents a unified, nonparametric output analysis method that extends bootstrap and influence functions for higher-order bias correction and variance reduction.

Findings

Enhanced robustness of confidence intervals with higher coverage probability

Effective bias correction using extended bootstrap and influence functions

Variance reduction achieved through control variates

Abstract

In predictive modeling with simulation or machine learning, it is critical to accurately assess the quality of estimated values through output analysis. In recent decades output analysis has become enriched with methods that quantify the impact of input data uncertainty in the model outputs to increase robustness. However, most developments are applicable assuming that the input data adheres to a parametric family of distributions. We propose a unified output analysis framework for simulation and machine learning outputs through the lens of Monte Carlo sampling. This framework provides nonparametric quantification of the variance and bias induced in the outputs with higher-order accuracy. Our new bias-corrected estimation from the model outputs leverages the extension of fast iterative bootstrap sampling and higher-order influence functions. For the scalability of the proposed estimation methods, we devise budget-optimal rules and leverage control variates for variance reduction. Our theoretical and numerical results demonstrate a clear advantage in building more robust confidence intervals from the model outputs with higher coverage probability.