Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models

TL;DR

This paper emphasizes that tokenization is a fundamental design choice in large language models, advocating for integrated, context-aware approaches to improve fairness, efficiency, and adaptability.

Contribution

It redefines tokenization as a core modeling decision, proposing a co-design framework and standardized evaluation to enhance language model performance.

Findings

Tokenization significantly impacts model fairness and efficiency.

Current subword methods often misalign with linguistic structures.

A co-design approach can improve model adaptability across languages.

Abstract

Tokenization underlies every large language model, yet it remains an under-theorized and inconsistently designed component. Common subword approaches such as Byte Pair Encoding (BPE) offer scalability but often misalign with linguistic structure, amplify bias, and waste capacity across languages and domains. This paper reframes tokenization as a core modeling decision rather than a preprocessing step. We argue for a context-aware framework that integrates tokenizer and model co-design, guided by linguistic, domain, and deployment considerations. Standardized evaluation and transparent reporting are essential to make tokenization choices accountable and comparable. Treating tokenization as a core design problem, not a technical afterthought, can yield language technologies that are fairer, more efficient, and more adaptable.