An Information-Theoretic Perspective on LLM Tokenizers

TL;DR

This paper investigates how LLM tokenizers function as structured compressors, revealing how training scale affects entropy distribution, and introduces a compression-aware tokenizer variant to improve robustness and efficiency.

Contribution

It provides empirical insights into tokenizer behavior across scales and domains, benchmarks existing tokenizers, and proposes a new compression-aware BPE variant for better performance.

Findings

Tokenizer training redistributes entropy with scale

Tokenizers absorb short-range regularity but degrade under domain mismatch

Compression-aware BPE improves robustness and efficiency

Abstract

Large language model (LLM) tokenizers act as structured compressors: by mapping text to discrete token sequences, they determine token count (and thus compute and context usage) and the statistical structure seen by downstream models. Despite their central role in LLM pipelines, the link between tokenization, compression efficiency and induced structure is not well understood. We empirically demonstrate that tokenizer training scale redistributes entropy: as training data grows, the token stream becomes more diverse in aggregate (higher unigram entropy) yet markedly more predictable in-context (lower higher-order conditional entropies), indicating that tokenization absorbs substantial short-range regularity although these gains degrade under train-test domain mismatch. To ground these observations, we first benchmark i) pretrained GPT-family tokenizers as black-box compressors across various domains, and ii) learned tokenizers across configurations spanning vocabulary size, training scale, and domain. Next, we study tokenization as a transform for universal compression and introduce a compression-aware BPE variant. Finally, we adopt a channel lens and introduce capacity-utilization metrics to analyze tokenizer behaviour and outline implications for downstream modeling. Put together, our results expose various trade-offs between compression, induced structure, and robustness under domain shift, and motivate principled, compression-aware tokenizer design.