Investigating the Impact of Randomness on Reproducibility in Computer Vision: A Study on Applications in Civil Engineering and Medicine

TL;DR

This study examines how CUDA-induced randomness affects reproducibility in computer vision applications, revealing that it can cause performance variability up to 4.77%, with management strategies impacting runtime and performance.

Contribution

It is the first detailed investigation into CUDA-induced randomness in computer vision, quantifying its impact on reproducibility across multiple datasets.

Findings

CUDA randomness causes up to 4.77% performance variation

Managing randomness can increase runtime or reduce performance

Disadvantages of managing randomness are less severe than previously thought

Abstract

Reproducibility is essential for scientific research. However, in computer vision, achieving consistent results is challenging due to various factors. One influential, yet often unrecognized, factor is CUDA-induced randomness. Despite CUDA's advantages for accelerating algorithm execution on GPUs, if not controlled, its behavior across multiple executions remains non-deterministic. While reproducibility issues in ML being researched, the implications of CUDA-induced randomness in application are yet to be understood. Our investigation focuses on this randomness across one standard benchmark dataset and two real-world datasets in an isolated environment. Our results show that CUDA-induced randomness can account for differences up to 4.77% in performance scores. We find that managing this variability for reproducibility may entail increased runtime or reduce performance, but that disadvantages are not as significant as reported in previous studies.