Random Text, Zipf's Law, Critical Length,and Implications for Large Language Models

TL;DR

This paper presents a simple non-linguistic model of text that derives Zipf's law and critical word length, providing a null hypothesis for understanding word statistics in natural language and large language models.

Contribution

It offers a unified, explicit mathematical derivation linking word length distribution, vocabulary growth, and rank-frequency law in a purely combinatorial model.

Findings

Word lengths follow a geometric distribution based on space probability.

A critical word length k* marks the transition from frequent to rare words.

Zipf's law emerges naturally from combinatorics without linguistic assumptions.

Abstract

We study a deliberately simple, fully non-linguistic model of text: a sequence of independent draws from a finite alphabet of letters plus a single space symbol. A word is defined as a maximal block of non-space symbols. Within this symbol-level framework, which assumes no morphology, syntax, or semantics, we derive several structural results. First, word lengths follow a geometric distribution governed solely by the probability of the space symbol. Second, the expected number of words of a given length, and the expected number of distinct words of that length, admit closed-form expressions based on a coupon-collector argument. This yields a critical word length k* at which word types transition from appearing many times on average to appearing at most once. Third, combining the exponential growth of the number of possible strings of length k with the exponential decay of the probability of each string, we obtain a Zipf-type rank-frequency law p(r) proportional to r^{-alpha}, with an exponent determined explicitly by the alphabet size and the space probability. Our contribution is twofold. Mathematically, we give a unified derivation linking word lengths, vocabulary growth, critical length, and rank-frequency structure in a single explicit model. Conceptually, we argue that this provides a structurally grounded null model for both natural-language word statistics and token statistics in large language models. The results show that Zipf-like patterns can arise purely from combinatorics and segmentation, without optimization principles or linguistic organization, and help clarify which phenomena require deeper explanation beyond random-text structure.