The exact power law and Pascal pyramid

TL;DR

This paper proves the existence of a power law limit for the probability distribution of words generated from a probabilistic alphabet with a stop symbol, providing an explicit formula involving entropy.

Contribution

It establishes the exact power law behavior of word probabilities and derives an explicit formula for the limit constant, extending previous weaker results.

Findings

The probability of the r-th most likely word follows a power law asymptotically.

The limit constant can be expressed via the entropy of a transformed distribution.

The result holds under the condition of irrationality in the log-probability ratios.

Abstract

Let $\omega_0, \omega_1,\ldots, \omega_n$ be a full set of outcomes (letters, symbols) and let positive $p_i$, $i=0,\ldots,n$, be their probabilities ($\sum_{i=0}^n p_i=1$). Let us treat $\omega_0$ as a stop symbol; it can occur in sequences of symbols (we call them words) only once, at the very end. The probability of a word is defined as the product of probabilities of its letters. We consider the list of all possible words sorted in the non-increasing order of their probabilities. Let $p(r)$ be the probability of the $r$th word in this list. We prove that if at least one of ratios $\log p_i/\log p_j$, $i,j\in\{ 1,\ldots,n\}$, is irrational, then the limit $\lim_{r\to\infty} p(r)/r^{1/\gamma}$ exists and differs from zero; here $\gamma$ is the root of the equation $\sum_{i=1}^n p_i^\gamma=1$. Some weaker results were established earlier. We are first to write an explicit formula for this limit constant at the power function; it can be expressed (rather easily) in terms of the entropy of the distribution~$(p_1^\gamma,\ldots,p_n^\gamma)$.