How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers
Jonathan Roberts, Kai Han, Samuel Albanie

TL;DR
This paper empirically analyzes how tokenization varies across different models and text domains, revealing significant discrepancies that challenge common assumptions about token length and consistency in large language models.
Contribution
It provides a comprehensive empirical study quantifying tokenization variation across models and text types, offering new insights into tokenization practices.
Findings
Tokenization varies significantly across models and text domains.
Common heuristics about token lengths are overly simplistic.
Insights improve understanding of tokenization in large language models.
Abstract
Frontier LLMs are increasingly utilised across academia, society and industry. A commonly used unit for comparing models, their inputs and outputs, and estimating inference pricing is the token. In general, tokens are used as a stable currency, assumed to be broadly consistent across tokenizers and contexts, enabling direct comparisons. However, tokenization varies significantly across models and domains of text, making naive interpretation of token counts problematic. We quantify this variation by providing a comprehensive empirical analysis of tokenization, exploring the compression of sequences to tokens across different distributions of textual data. Our analysis challenges commonly held heuristics about token lengths, finding them to be overly simplistic. We hope the insights of our study add clarity and intuition toward tokenization in contemporary LLMs.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsStock Market Forecasting Methods · Blockchain Technology Applications and Security · Art History and Market Analysis
