Accurate simulation of operating system updates in neuroimaging using Monte-Carlo arithmetic

TL;DR

This paper introduces a Monte-Carlo arithmetic framework called 'fuzzy libmath' to accurately simulate the variability in neuroimaging results caused by operating system updates, aiding in quantifying numerical uncertainty.

Contribution

It presents a novel method to reproduce OS update-induced variability in neuroimaging pipelines using Monte-Carlo arithmetic, enhancing reproducibility analysis.

Findings

FL-perturbed pipelines mimic OS update variability

Between-subject differences are preserved despite perturbations

Numerical precision in pre-processed images is relatively low

Abstract

Operating system (OS) updates introduce numerical perturbations that impact the reproducibility of computational pipelines. In neuroimaging, this has important practical implications on the validity of computational results, particularly when obtained in systems such as high-performance computing clusters where the experimenter does not control software updates. We present a framework to reproduce the variability induced by OS updates in controlled conditions. We hypothesize that OS updates impact computational pipelines mainly through numerical perturbations originating in mathematical libraries, which we simulate using Monte-Carlo arithmetic in a framework called "fuzzy libmath" (FL). We applied this methodology to pre-processing pipelines of the Human Connectome Project, a flagship open-data project in neuroimaging. We found that FL-perturbed pipelines accurately reproduce the variability induced by OS updates and that this similarity is only mildly dependent on simulation parameters. Importantly, we also found between-subject differences were preserved in both cases, though the between-run variability was of comparable magnitude for both FL and OS perturbations. We found the numerical precision in the HCP pre-processed images to be relatively low, with less than 8 significant bits among the 24 available, which motivates further investigation of the numerical stability of components in the tested pipeline. Overall, our results establish that FL accurately simulates results variability due to OS updates, and is a practical framework to quantify numerical uncertainty in neuroimaging.