Zipf's law unzipped

TL;DR

This paper introduces a universal Random Group Formation model that explains Zipf's law across diverse phenomena by predicting group size distributions based on minimal information, without system-specific assumptions.

Contribution

It presents a Bayesian-based RGF model that predicts group size distributions with a specific power-law form, linking the exponent to basic data characteristics.

Findings

RGF accurately predicts data distributions across systems

The power-law exponent γ typically ranges from 1 to 2

γ systematically depends on total data size

Abstract

Why does Zipf's law give a good description of data from seemingly completely unrelated phenomena? Here it is argued that the reason is that they can all be described as outcomes of a ubiquitous random group division: the elements can be citizens of a country and the groups family names, or the elements can be all the words making up a novel and the groups the unique words, or the elements could be inhabitants and the groups the cities in a country, and so on. A Random Group Formation (RGF) is presented from which a Bayesian estimate is obtained based on minimal information: it provides the best prediction for the number of groups with $k$ elements, given the total number of elements, groups, and the number of elements in the largest group. For each specification of these three values, the RGF predicts a unique group distribution $N(k)\propto \exp(-bk)/k^{\gamma}$, where the power-law index $\gamma$ is a unique function of the same three values. The universality of the result is made possible by the fact that no system specific assumptions are made about the mechanism responsible for the group division. The direct relation between $\gamma$ and the total number of elements, groups, and the number of elements in the largest group, is calculated. The predictive power of the RGF model is demonstrated by direct comparison with data from a variety of systems. It is shown that $\gamma$ usually takes values in the interval $1\leq\gamma\leq 2$ and that the value for a given phenomena depends in a systematic way on the total size of the data set. The results are put in the context of earlier discussions on Zipf's and Gibrat's laws, $N(k)\propto k^{-2}$ and the connection between growth models and RGF is elucidated.