A path to natural language through tokenisation and transformers

TL;DR

This paper investigates how byte-pair encoding (BPE) tokenisation influences the statistical properties of language models, showing that deeper BPE levels align model entropy with natural language laws like Zipf's, thus clarifying BPE's role in language representation.

Contribution

It provides a theoretical and empirical analysis of BPE's effect on language statistics, linking tokenisation depth to natural language regularities and model entropy.

Findings

Deeper BPE levels induce Zipfian distribution in token frequencies.

Model entropy predictions align with Zipf's law as BPE depth increases.

Deeper tokenisation reduces local token dependencies, approaching IID conditions.

Abstract

Natural languages exhibit striking regularities in their statistical structure, including notably the emergence of Zipf's and Heaps' laws. Despite this, it remains broadly unclear how these properties relate to the modern tokenisation schemes used in contemporary transformer models. In this note, we analyse the information content (as measured by the Shannon entropy) of various corpora under the assumption of a Zipfian frequency distribution, and derive a closed-form expression for the slot entropy expectation value. We then empirically investigate how byte--pair encoding (BPE) transforms corpus statistics, showing that recursive applications of BPE drive token frequencies toward a Zipfian power law while inducing a characteristic growth pattern in empirical entropy. Utilizing the ability of transformers to learn context dependent token probability distributions, we train language models on corpora tokenised at varying BPE depths, revealing that the model predictive entropies increasingly agree with Zipf-derived predictions as the BPE depth increases. Attention-based diagnostics further indicate that deeper tokenisation reduces local token dependencies, bringing the empirical distribution closer to the weakly dependent (near IID) regime. Together, these results clarify how BPE acts not only as a compression mechanism but also as a statistical transform that reconstructs key informational properties of natural language.