Hypothesis testing of scientific Monte Carlo calculations

TL;DR

This paper advocates for integrating statistical hypothesis testing into the standard validation toolkit for scientific Monte Carlo simulations to improve detection of numerical issues and bugs.

Contribution

It demonstrates how hypothesis testing can be adapted for Monte Carlo methods, providing automatic, reliable tests to identify common simulation problems.

Findings

Hypothesis tests can detect numerical problems in Monte Carlo simulations.

Automated testing improves robustness and reproducibility of stochastic scientific calculations.

The approach is applicable to various scientific Monte Carlo applications.

Abstract

The steadily increasing size of scientific Monte Carlo simulations and the desire for robust, correct, and reproducible results necessitates rigorous testing procedures for scientific simulations in order to detect numerical problems and programming bugs. However, the testing paradigms developed for deterministic algorithms have proven to be ill suited for stochastic algorithms. In this paper we demonstrate explicitly how the technique of statistical hypothesis testing, which is in wide use in other fields of science, can be used to devise automatic and reliable tests for Monte Carlo methods, and we show that these tests are able to detect some of the common problems encountered in stochastic scientific simulations. We argue that hypothesis testing should become part of the standard testing toolkit for scientific simulations.