Correct ordering in the Zipf-Poisson ensemble

TL;DR

This paper analyzes the ordering of word frequencies modeled by a Zipf-Poisson ensemble, establishing probabilistic bounds for correct ordering of top-ranked words as the total count grows large.

Contribution

It provides explicit probabilistic bounds for the correct ordering of top elements in a Zipf-Poisson model, including practical estimates for real-world data like the British National Corpus.

Findings

First n' words are correctly ordered with high probability up to a specific growth rate.

The exact rate of N^{1/( extalpha+2)} cannot be achieved.

In a large corpus, the top 72 words are correctly ordered with high probability.

Abstract

We consider a Zipf--Poisson ensemble in which $X_i\sim\poi(Ni^{-\alpha})$ for $\alpha>1$ and $N>0$ and integers $i\ge 1$. As $N\to\infty$ the first $n'(N)$ random variables have their proper order $X_1>X_2>...>X_{n'}$ relative to each other, with probability tending to 1 for $n'$ up to $(AN/\log(N))^{1/(\alpha+2)}$ for an explicit constant $A(\alpha)\ge 3/4$. The rate $N^{1/(\alpha+2)}$ cannot be achieved. The ordering of the first $n'(N)$ entities does not preclude $X_m>X_{n'}$ for some interloping $m>n'$. The first $n"$ random variables are correctly ordered exclusive of any interlopers, with probability tending to 1 if $n"\le (BN/\log(N))^{1/(\alpha+2)}$ for $B<A$. For a Zipf--Poisson model of the British National Corpus, which has a total word count of $100{,}000{,}000$, our result estimates that the 72 words with the highest counts are properly ordered.