Scaling Laws in Human Language

TL;DR

This paper investigates the applicability of Zipf's law across different languages, proposing a model that accounts for finite vocabulary effects, and refines understanding of language scaling laws through experiments and analysis.

Contribution

It introduces a new model explaining language frequency distributions considering vocabulary size, and clarifies the conditions under which Zipf's law holds or diverges.

Findings

Zipf's law holds for some languages but not others due to vocabulary effects.

The frequency distribution follows a power law with an exponent of 1, diverging at this point.

The growth of distinct words with text length exhibits three stages: linear, logarithmic, and saturation.

Abstract

Zipf's law on word frequency is observed in English, French, Spanish, Italian, and so on, yet it does not hold for Chinese, Japanese or Korean characters. A model for writing process is proposed to explain the above difference, which takes into account the effects of finite vocabulary size. Experiments, simulations and analytical solution agree well with each other. The results show that the frequency distribution follows a power law with exponent being equal to 1, at which the corresponding Zipf's exponent diverges. Actually, the distribution obeys exponential form in the Zipf's plot. Deviating from the Heaps' law, the number of distinct words grows with the text length in three stages: It grows linearly in the beginning, then turns to a logarithmical form, and eventually saturates. This work refines previous understanding about Zipf's law and Heaps' law in language systems.