On the reproducibility of discrete-event simulation studies in health research: an empirical study using open models

TL;DR

This study empirically evaluates the reproducibility of eight healthcare discrete-event simulation models, identifying key barriers and proposing actionable recommendations to improve transparency and reproducibility in health research.

Contribution

It provides the first empirical assessment of reproducibility challenges in healthcare DES models and offers practical guidelines for enhancing reproducibility practices.

Findings

50% of models were fully reproducible

Troubleshooting took up to 28 hours per model

Key barriers included missing code and lack of open licenses

Abstract

Reproducibility of computational research is critical for ensuring transparency, reliability and reusability. Challenges with computational reproducibility have been documented in several fields, but healthcare discrete-event simulation (DES) models have not been thoroughly examined in this context. This study assessed the computational reproducibility of eight published healthcare DES models (Python or R), selected to represent diverse contexts, complexities, and years of publication. Repositories and articles were also assessed against guidelines and reporting standards, offering insights into their relationship with reproducibility success. Reproducing results required up to 28 hours of troubleshooting per model, with 50% fully reproduced and 50% partially reproduced (12.5% to 94.1% of reported outcomes). Key barriers included the absence of open licences, discrepancies between reported and coded parameters, and missing code to produce model outputs, run scenarios, and generate tables and figures. Addressing these issues would often require relatively little effort from authors: adding an open licence and sharing all materials used to produce the article. Actionable recommendations are proposed to enhance reproducibility practices for simulation modellers and reviewers.