Transformers in Pseudo-Random Number Generation: A Dual Perspective on Theory and Practice

TL;DR

This paper investigates the use of Transformer models for pseudo-random number generation, demonstrating their theoretical ability to simulate traditional PRNGs and validating their statistical randomness through experiments.

Contribution

It introduces a novel approach of using decoder-only Transformers for PRNGs, showing they can simulate classical generators and produce statistically random numbers.

Findings

Transformers can simulate LCG and Mersenne Twister PRNGs.

Transformer-based PRNGs pass most NIST randomness tests.

They are capable of resisting certain prediction attacks.

Abstract

Pseudo-random number generators (PRNGs) are high-nonlinear processes, and they are key blocks in optimization of Large language models. Transformers excel at processing complex nonlinear relationships. Thus it is reasonable to generate high-quality pseudo-random numbers based on transformers. In this paper, we explore this question from both theoretical and practical perspectives, highlighting the potential benefits and implications of Transformer in PRNGs. We theoretically demonstrate that decoder-only Transformer models with Chain-of-Thought can simulate both the Linear Congruential Generator (LCG) and Mersenne Twister (MT) PRNGs. Based on this, we conclude that the log-precision decoder-only Transformer can represent non-uniform $\text{AC}^0$. Our simulative theoretical findings are validated through experiments. The random numbers generated by Transformer-based PRNGs successfully pass the majority of NIST tests, whose heat maps exhibit clear statistical randomness. Finally, we assess their capability in prediction attacks.