Statistical Quality and Reproducibility of Pseudorandom Number Generators in Machine Learning technologies

TL;DR

This study evaluates the statistical quality and reproducibility of pseudorandom number generators used in popular machine learning frameworks, revealing potential weaknesses and differences between native and integrated implementations.

Contribution

It provides a comprehensive comparison of PRNGs in ML frameworks against their original implementations using rigorous testing, highlighting issues in statistical robustness and implementation differences.

Findings

Some PRNGs fail statistical tests despite being labeled 'crush-resistant'

Differences observed between native and framework-integrated PRNG versions

Framework-integrated PRNGs may have compromised statistical quality

Abstract

Machine learning (ML) frameworks rely heavily on pseudorandom number generators (PRNGs) for tasks such as data shuffling, weight initialization, dropout, and optimization. Yet, the statistical quality and reproducibility of these generators-particularly when integrated into frameworks like PyTorch, TensorFlow, and NumPy-are underexplored. In this paper, we compare the statistical quality of PRNGs used in ML frameworks (Mersenne Twister, PCG, and Philox) against their original C implementations. Using the rigorous TestU01 BigCrush test suite, we evaluate 896 independent random streams for each generator. Our findings challenge claims of statistical robustness, revealing that even generators labeled ''crush-resistant'' (e.g., PCG, Philox) may fail certain statistical tests. Surprisingly, we can observe some differences in failure profiles between the native and framework-integrated versions of the same algorithm, highlighting some implementation differences that may exist.