The explicit formulae for the distributions of nonoverlapping words and its applications to statistical tests for pseudo random numbers

TL;DR

This paper derives explicit formulas for the distributions of nonoverlapping words in Bernoulli models, enabling improved statistical tests for pseudo-random number generators and analyzing their power.

Contribution

It provides explicit finite-dimensional distribution formulas for nonoverlapping words, extending previous generating function approaches, and applies these to enhance statistical testing methods.

Findings

Explicit formulas for nonoverlapping word distributions in Bernoulli models.

Statistically significant power difference between tests based on word counts.

Application of formulas to evaluate pseudo-random number generators.

Abstract

The distributions of the number of occurrences of words (the distributions of words for short) play key roles in information theory, statistics, probability theory, ergodic theory, computer science, and DNA analysis. Bassino et al. 2010 and Regnier et al. 1998 showed generating functions of the distributions of words for all sample sizes. Robin et al. 1999 presented generating functions of the distributions for the return time of words and demonstrated a recurrence formula for these distributions. These generating functions are rational functions; except for simple cases, it is difficult to expand them into power series. In this paper, we study finite-dimensional generating functions of the distributions of nonoverlapping words for each fixed sample size and demonstrate the explicit formulae for the distributions of words for the Bernoulli models. Our results are generalized to nonoverlapping partial words. We study statistical tests that depend on the number of occurrences of words and the number of block-wise occurrences of words, respectively. We demonstrate that the power of the test that depends on the number of occurrences of words is significantly large compared to the other one. Finally, we apply our results to statistical tests for pseudo random numbers.