A test against trend in random sequences

TL;DR

This paper introduces a modified Kendall's tau-test that uses sequences with repetitions, including binary sequences, to evaluate randomness, expanding the applicability of the test to more general sequences.

Contribution

It proposes a new version of Kendall's tau-test that accommodates sequences with repeated elements, broadening the scope of randomness testing methods.

Findings

The modified test can evaluate the randomness of binary and other sequences.

It extends Kendall's tau-test to sequences with repetitions.

The method is applicable to a wider class of sequences, including those with repeated elements.

Abstract

We study a modification of Kendall's tau-test, replacing his permutations of n different numbers by sequences of length n, where repetition is allowed. In particular, binary sequences are included. Random sequences can be tested.

Equations

$$x_1,x_2,...,x_n$$

$$x_i>x_j\text{with}i>j$$

$$S(x_1,...,x_n)=S^{+}(x_1,...,x_n)-S^{-}(x_1,...,x_n).$$

$$\frac{n!}{(2!{)}^{p_2}(3!{)}^{p_3}\cdots (r!{)}^{p_r}}(p_1+2p_2+...+r{p}_r=n)$$

$$\mu (n)=0.$$

$${\sigma }^2(n)=\frac{\ell -1}{\ell }\frac{n(n-1)}{2}+\frac{{\ell }^2-1}{{\ell }^2}\frac{n(n-1)(n-2)}{9}$$

$${\mu }_4(n)$$

$$+\frac{{\ell }^2-1}{{\ell }^4}\frac{252n^3+507n^2-3623n+3652}{900}n(n-1)$$

$$-\frac{{\ell }^2-1}{{\ell }^3}\frac{2n^3+3n^2-5n-15}{6}n(n-1)$$

$$+\frac{\ell -1}{{\ell }^3}\frac{n^2+11n-25}{2}n(n-1).$$

$${\mu }_3(n)=0,{\mu }_5(n)=0,{\mu }_7(n)=0,....$$

$$\frac{1}{\sigma (n)\sqrt{2\pi }}{\int }_{S_1}^{S_2}{e}^{-x^2/2{\sigma }^2(n)}dx,$$

$$\underset{n\to \infty }{\lim }\frac{{\mu }_{2k}(n)}{(\sigma (n){)}^{2k}}=1\cdot 2\cdot 5\cdots (2k-1)$$

$$f(x)\,=\,\begin{cases}\frac{1}{2^{2\nu}}\,\underset{k=1}{\overset{k=\nu}{\prod}}\bigl{(}x^{2k-1}+2+x^{1-2k}\bigr{)},\qquad n=2\nu\\ \frac{2}{2^{2\nu+1}}\,\underset{k=1}{\overset{k=\nu}{\prod}}\bigl{(}x^{k}+x^{-k}\bigr{)}^{2},\qquad\qquad\,\,\,n=2\nu+1\end{cases}$$

$${\sigma }^2(n)=\frac{n(n^2-1)}{3}pq$$

$${\mu }_4(n)$$

$$+\frac{n(n^2-1)(3n^2-7)}{15}pq(p-q{)}^2.$$

$$0000010010021102$$

$$001-2011-210101110$$

$$\underline{-2-10+1+2}$$

$$20402$$

$$2$$

$$121$$

$$20402$$

$$121242121$$

$$2040608060402$$

$$1212445458545442121$$

$$\overline{121}$$

$$121$$

$$121121$$

$$\overline{121242121}$$

$$121242121$$

$$121242121121242121$$

$$\overline{1212445458545442121}$$

$$\overline{020}$$

$$020$$

$$020020$$

$$\overline{0204020}$$

$$0204020$$

$$02040200204020$$

$$\overline{020406080604020}$$

$$020406080604020$$

Full text

A TEST AGAINST TREND IN RANDOM SEQUENCES

Peter Lindqvist

(Department of Mathematical Sciences

Norwegian University of Science and Technology

NO–7491 Trondheim, Norway)

Abstract:111AMS classification 62G60 We study a modification of Kendall’s $\tau$, replacing his permutations of $n$ different numbers by sequences of length $n$. Thus repetition is allowed. In particular, binary sequences are studied.

1 Introduction

The basic tool in Kendall’s $\tau$-test is the “score” $S$. Suppose that $\ell$ digits $0,1,...,\ell-1$ satisfying the transitive relations $0<1<...<\ell-1$ are given and consider all the $\ell^{n}$ possible sequences

[TABLE]

of length $n$ that can be formed by the aid of these digits. Let $S^{+}(x_{1},...,x_{n})$ denote the number of true inequalities

[TABLE]

in the sequence (1). Analogously, $S^{-}(x_{1},...,x_{n})$ counts the number of all valid inequalities $x_{i}<x_{j}$ with $i>j$. Define

[TABLE]

For example, $S=5-11=-6$ for the sequence $0\,1\,1\,2\,0\,2\,1.$ Originally222The coefficient $\tau$ was considered by Greiner (1909) and Esscher (1924). The coefficient was rediscovered by Kendall (1938)., Kendall considered the distribution of $S$ among the $\ell!$ permutations of the digits $\,0,1,...,\ell-1\,$ (then $n=\ell$), and so far as I know333However, see [L1]. the generalizations of Kendall’s $\tau$-test rely upon the distribution of $S$ among all

[TABLE]

possible sequences (1) consisting of $p_{1}$ digits occurring only once, $p_{2}$ pairs, $p_{3}$ triplets, and so on. Here the numbers of ties, i.e., $p_{1},p_{2},...,p_{r}$, are regarded as fixed. See [S]. For permutations $S^{+}$ has been thoroughly investigated in [M].

We shall study a different situation, arising for example in connexion with the testing of sequences of random digits. In this setting the number of ties cannot be regarded as fixed a priori. Thus we are led to study the distribution of $S$ among all $\ell^{n}$ sequences (1). As we shall learn,* this distribution approaches normality, as $n\to\infty.$*

Having in mind applications for a certain kind of sampling, we have considered the binary case $\ell=2$ also when the probability for a [math] is $p$ and the probability for a $1$ is $q$, where $p+q=1.$ See Section 5.

Finally, we mention that the distribution for $S$ in our setting of the problem is, in certain respects even simpler than the version considered by Kendall [K1], Sillitto [S], and Silverstone [S2].

2 The Basic Results

The mean value for $S$ taken over all sequences (1) is zero by symmetry:

[TABLE]

When all the sequences are equiprobable, the variance is

[TABLE]

and the fourth central moment is

[TABLE]

By symmetry

[TABLE]

It is interesting to observe that for $n$ fixed the moments approach those given by Kendall in [K1], as $\ell\to\infty.$ Formula (4) is derived in Section 6, but the corresponding calculations for (2) are, to say the least, a laborious task and so $\mu_{4}(n)$ is given without proof, when $\ell\geq 3.$

As $n$ grows, the distribution for $S$ tends towards normality in the sense that the frequency between the values $S_{1}$ and $S_{2}$ tends to

[TABLE]

where the standard deviation is $\sigma(n)=\sqrt{\mu_{2}(n)}.$ This follows from the Second Limit Theorem, since

[TABLE]

and the odd moments are zero. It is easy to prove (6) for small values of $\ell$, but the probability function for $S$ becomes soon too complicated, as $\ell$ grows. Therefore we shall prove (6) only for the binary case $\ell=2$, see Section 4.

In the binary case the probability generating function for $S$ is

[TABLE]

and so the characteristic function $\phi(\theta)\,=\,f(e^{i\theta})$ reduces to the simple expression (10). Our proof for (6), when $\ell=2$ is based on $\phi(\theta)$. Furthermore, the distribution for $S$ can be rapidly calculated via a suitable interpretation of (7).

In the general binary case, when the probability that $x_{i}=0$ is $p$ and that $x_{i}=1$ is $q$ in (1), $i=1,2,...,n,\,\,p+q=1,$ the corresponding probability function is given by (18) and the characteristic function by (19). Now again $\mu(n)=0$, and

[TABLE]

and the fourth moment is

[TABLE]

All odd moments are zero and the asymptotic normality (6) holds even for $p\neq q$.

3 The Probability Function

Consider the binary case $\ell=2$ with equiprobable sequences (1). Direct calculation of $S$ for $n=3$ yields

[TABLE]

and so we can write

[TABLE]

and in this manner the distribution for $S$ can be tabulated. The results are displayed below:

[TABLE]

The fundamental observation is that the table can be constructed via the following kind of figurates. For even $n$ we have

[TABLE]

and so on. (The overlined sequences display $n=2,\,4$ and $6$.) For odd $n$ the table looks like

[TABLE]

and the next row, corresponding to $n=7$, becomes

[TABLE]

(The first and last zeros in a row are void.) In an obvious interpretation the above process reads

[TABLE]

for odd $n$ and

[TABLE]

for even $n$. This leads to the probability generating function (7).

A simple proof for the probability generating function $f(x)$ comes from considering the binary sequence

[TABLE]

Then we have

[TABLE]

and the index $j_{k}$ appears exactly $(n-k)-(k-1)$ times and so its contribution to the score $S$ is

[TABLE]

Therefore

[TABLE]

for this sequence. Now $j_{k}$ is [math] or $1$ so that the generating function becomes

[TABLE]

Upon multiplication, the coefficient of $x^{t}$ indicates how many times $S=t$ among all possible sequences $j_{1},j_{2},...,j_{n}$. Dividing by the total number of sequences we arrive at the probability generating function $(\ref{f})$.

The characteristic function for $S$ is $\phi(\theta)\,=\,f(e^{i\theta}),\,\,i^{2}=-1.$ Euler’s formula yields

[TABLE]

By definition $\phi(0)=1$ and

[TABLE]

By symmetry $\phi^{\prime}(0)=0,\,\phi^{3}(0)=0$,…, so that all odd moments are zero and

[TABLE]

Direct calculations yield

[TABLE]

The arrangements in Section 4 will shorten such calculations.

4 Approach to Normality

In order to show that the distribution for $S$ approaches normality in the binary case, we shall prove (6). The dichotomy in formulae (10) forces us to separate the cases

[TABLE]

However, both cases are so similar that we shall write down only the odd case $n=2\nu+1.$ Then the characteristic function is

[TABLE]

and by logarithmic differentiation

[TABLE]

Denoting

[TABLE]

we obviously have

[TABLE]

In order to calculate the odd derivatives $A^{\prime}(0),\,A^{\prime\prime\prime}(0),...$ we use the expansion

[TABLE]

where $\mathrm{B}_{0}=1,\mathrm{B}_{2}=1/6,\mathrm{B}_{4}=-1/30,...$ are the Bernoulli numbers. Thus

[TABLE]

for $|\theta|<\pi/2\nu.$ We deduce that

[TABLE]

where

[TABLE]

are well-known polynomials of degree $2k+1$.

According to (11) and (12) we obtain444The connexion with cumulants is obvious, since $\frac{d}{d\theta}\log|\phi(\theta)|=A(\theta).$ by Leibniz rule

[TABLE]

In passing, we calculate

[TABLE]

for odd $n$. For even $n$ we shall arrive at the same formulae. We have obtained that

[TABLE]

This shows that (6) holds at least for $k=1$ and $k=2$. For general $k$ we use induction.

To this end, notice that at the point $\theta=0$ we have

[TABLE]

or, more conveniently,

[TABLE]

According to (14) and (15), where $n=2\nu+1$ is odd, we have

[TABLE]

since $A^{(2j-1)}\approx\nu^{2j+1}$ and $(\phi^{\prime\prime})^{j}\approx\nu^{3j}.$ But now (17) shows that, if

[TABLE]

holds for $m=1,2,...,k-1,$ then also

[TABLE]

In other words, the desired conclusion (6) follows by induction with respect to $k$. This concludes our proof of the asymptotic normality.

5 Binary Sequences not Equiprobable

Consider again all $2^{n}$ sequences of length $n$ consisting merely of [math]’s and $1$’s. But assume now that the probability for a [math] is $P(0)=p$ and the probability for an $1$ is $P(1)=q.$ Here $p+q=1.$ For example, the sequence $0\,1\,1\,0\,1\,1\,1$ has probability $p^{2}q^{5}$. The figurates in Table I and Table II (at the end) are constructed via (6) below. The simple rule for the formation of these figurates is condensed in the formulae

[TABLE]

for the probability generating function. (Of course, the factor $p+q$ outside the product is $1$, but it is included to match Table II.)

The characteristic function $\phi(\theta)=f(e^{i\theta})$ is

[TABLE]

For $p=q=1/2$ we again obtain the expressions in Section 2.

Let us consider the case $n=2\nu+1$, the calculations for even $n$ being similar. Now

[TABLE]

with an obvious abbreviation. The well-known expansion

[TABLE]

converges for

[TABLE]

Having treated the case $p=q=1/2$ in the previous sections, we assume that $p\neq q$ here. Then

[TABLE]

and so we obtain

[TABLE]

Using the Maclaurin series for $\sin(2km\theta)$, we arrive at the formula

[TABLE]

where some arrangements have been done. The corresponding convergence investigations are quite straightforward.

By (20), $A(0)=0,\,A^{\prime\prime}(0)=0,\,A^{(4)}(0)=0,\dots,$ and

[TABLE]

(This expansion diverges for $\varrho=-1$, i. e. for $p=q$.) Here the infinite sum is easily calculated as the differentiated geometric series

[TABLE]

A calculation yields

[TABLE]

and using

[TABLE]

we arrive at (8) and (2). —The corresponding calculations for even $n$ yield the same final result.

An analogous investigation as that in Section 4, but now based on (21), shows the approach to normality also for $p\neq q$. The difference is merely technical.

6 The Variance (with General $\ell$).

Consider again all sequences $x_{1},x_{2},\dots,x_{n}$ that can be formed of the digits $0,1,2,\dots,\ell$. Let $P_{n}(t;j_{0},j_{1},\dots,j_{\ell-1})$ count the number of those sequences consisting of $j_{0}\,\,0$’s, $j_{1}\,\,1$’s,…, $j_{\ell-1}\,\,\ell$’s,  $j_{0}+j_{1}+\dots+j_{\ell-1}=n$, for which $S=t$. For example $P_{6}(1;3,3)=3,\,P_{6}(4;2,4)=2,\,P_{9}(27;3,3,3)=1$, and $P_{9}(t;3,3,3)=0$ when $t\geq 28$.

Constructing the sequence $x_{1},x_{2},\dots,x_{n},x_{n+1}$ from $x_{1},x_{2},\dots,x_{n}$ we arrive at the fundamental recursive rule

[TABLE]

where now $i_{0}+i_{1}+\dots+i_{\ell-1}=n+1.$ In passing, we notice that applying the recursive formula twice we obtain for $\ell=2$ that

[TABLE]

Repeated use of this identity describes exactly how the configurations in Tables I and II are built up.

Let us return to the recursive rule (6). For the calculation of the variance $\mu_{2}(n)$ we assume that all sequences $x_{1},x_{2},\dots,x_{n}$ of length $n$ are equiprobable. Then

[TABLE]

where the sum is taken over all the $\ell^{n}$ possible sequences. The auxiliary quantities

[TABLE]

satisfy according to the recursive rule (6) the formula

[TABLE]

where $i_{0}+i_{1}+\dots+i_{\ell-1}=n+1.$ Here the first inner sum over $t$ is merely $B_{n}(i_{0},i_{1},\dots,i_{k-1},i_{k}-1,i_{k+1},\dots,i_{\ell-1})$ and the second inner sum is zero by symmetry. In the last inner sum $\sum_{t}P_{n}$ is a certain number of combinations.

Therefore the above formula reduces to the simple expression

[TABLE]

Here

[TABLE]

is the usual multinomial coefficient. Summing all equations with $i_{0}+i_{1}+\cdots+i_{\ell-1}=n+1$ and noting that

[TABLE]

we obtain

[TABLE]

The parenthesis in the sum can be written as the sum of products $\pm i_{\alpha}i_{\beta},$  $\ell-1$ of which are of the form $i_{\alpha}^{2},\,\,k^{2}+(\ell-k-1)^{2}$ of which are of the form $+i_{\alpha}i_{\beta}\,\,(\alpha\neq\beta)$, and $2k(\ell-k-1)$ of which are negative. Using well-known identities like

[TABLE]

we finally obtain the equation

[TABLE]

Adding the $n$ first equations (24) and noting that $\mu_{2}(1)=0$, we reach the final result (4). This concludes our proof for the variance $\sigma^{2}(n)$.

The fourth moment $\mu_{4}$ in (2) is the result of a similar, although more tedious, calculation. However, a more effective method should be invented for higher moments. —Corresponding formulae for $S^{+}$ are given in [L1].

In passing we mention the formula

[TABLE]

$i_{0}+\cdots+i_{\ell-1}=n$. Here the exceptional notation is understandable.

7 Edgeworth’s Approximation

The closedness to normality of the distribution for $S$ is good, when $l^{n}$ is large. However, if $l^{n}$ is not large, especially the tails of the distribution behave obstinately, so that the assumption of normality is somewhat inadequate for precisely those values of $S$ whose significance may be in doubt. Fortunately, numerical calculations indicate that a correction based on Edgeworth’s series gives an accurate approximation.

Let

[TABLE]

denote the normal distribution function. The approximation

[TABLE]

is obtained from Edgeworth’s series [C, page 229], terms containing $\mu_{6},\,\mu_{7},\dots$ being neglected.

It stands to reason that (25) is accurate, $\mu_{4}(n)$ and $\sigma(n)$ being calculated from (2) and (4), provided that $\ell^{n}$ is large, say $\ell^{n}>10^{6}$. —The dependence on $\ell$ is slightly puzzling. This is a point that requires further numerical investigation.