EFECT: A Method to Quantify the Reproducibility of Stochastic Simulations

TL;DR

EFECT is a new data-driven method that quantifies the reproducibility of stochastic simulation results, ensuring validation and credibility across various scientific fields.

Contribution

Introduces EFECT, a novel, broadly applicable method using empirical characteristic functions to measure and standardize reproducibility of stochastic simulations.

Findings

EFECT effectively quantifies reproducibility across diverse use cases.

It determines the number of simulation runs needed for reliable results.

EFECT enhances validation and credibility of stochastic models.

Abstract

Reproducibility is a fundamental requirement for validating scientific claims in computational research. Stochastic computational models are widely used in fields such as systems biology, financial modeling and environmental sciences. However, achieving reproducibility in stochastic simulations remains challenging, as each run can produce different outcomes. Existing infrastructure and software tools do not address independent reproduction of simulation results. Without independent reproducibility, results and conclusions lack credibility, as it remains unclear whether observed findings reflect model behavior or are artifacts of stochastic variation or an underpowered study. To bridge this gap, we introduce the Empirical Characteristic Function Equality Convergence Test (EFECT), a data-driven method to quantify the reproducibility of stochastic simulation results. EFECT employs empirical characteristic functions to compare reported results with those independently generated by assessing distributional inequality, termed EFECT error. Additionally, we establish the EFECT convergence point, a quantitative metric for determining the required number of simulation runs to achieve an EFECT error value of a priori significance. EFECT is applicable to all bounded, real-valued outputs, regardless of the model type or simulation method that produced them. We tested EFECT with over 40 use cases to demonstrate its broad applicability and effectiveness. EFECT standardizes stochastic simulation reproducibility, establishing a workflow that guarantees reliable results, supporting a wide range of stakeholders, and thereby enhancing validation of stochastic simulation studies, across a model's lifecycle. To promote standardization, we are developing the open-source software library libSSR in multiple programming languages for easy integration of EFECT.