HPC-driven computational reproducibility in numerical relativity codes: A use case study with IllinoisGRMHD

TL;DR

This study demonstrates that scientific results from the IllinoisGRMHD code in relativistic astrophysics can be reliably reproduced across different supercomputers and software versions, ensuring computational reproducibility in scientific computing.

Contribution

It provides a practical case study on achieving reproducibility in numerical relativity codes across diverse computing environments and software versions.

Findings

Reproduced results are consistent with original publication within round-off errors.

Reproducibility is maintained across different supercomputers (Expanse and Stampede2).

Updated code versions still reproduce previous results after bug fixes.

Abstract

Reproducibility of results is a cornerstone of the scientific method. Scientific computing encounters two challenges when aiming for this goal. Firstly, reproducibility should not depend on details of the runtime environment, such as the compiler version or computing environment, so results are verifiable by third-parties. Secondly, different versions of software code executed in the same runtime environment should produce consistent numerical results for physical quantities. In this manuscript, we test the feasibility of reproducing scientific results obtained using the IllinoisGRMHD code that is part of an open-source community software for simulation in relativistic astrophysics, the Einstein Toolkit. We verify that numerical results of simulating a single isolated neutron star with IllinoisGRMHD can be reproduced, and compare them to results reported by the code authors in 2015. We use two different supercomputers: Expanse at SDSC, and Stampede2 at TACC. By compiling the source code archived along with the paper on both Expanse and Stampede2, we find that IllinoisGRMHD reproduces results published in its announcement paper up to errors comparable to round-off level changes in initial data parameters. We also verify that a current version of IlliinoisGRMHD reproduces these results once we account for bug fixes which has occurred since the original publication