Toward a Theory of Tokenization in LLMs

TL;DR

This paper provides a theoretical investigation into how tokenization enhances transformer models' ability to learn complex sequence distributions, demonstrating that proper tokenization allows near-optimal modeling of Markovian data.

Contribution

It offers a theoretical analysis showing that tokenization enables transformers to effectively model higher-order Markov processes, explaining its practical importance.

Findings

Transformers without tokenization tend to learn unigram models.

With tokenization, transformers can model sequences near-optimally.

Tokenization significantly improves the modeling of Markovian data.

Abstract

While there has been a large body of research attempting to circumvent tokenization for language modeling (Clark et al., 2022; Xue et al., 2022), the current consensus is that it is a necessary initial step for designing state-of-the-art performant language models. In this paper, we investigate tokenization from a theoretical point of view by studying the behavior of transformers on simple data generating processes. When trained on data drawn from certain simple $k^{\text{th}}$-order Markov processes for $k > 1$, transformers exhibit a surprising phenomenon - in the absence of tokenization, they empirically fail to learn the right distribution and predict characters according to a unigram model (Makkuva et al., 2024). With the addition of tokenization, however, we empirically observe that transformers break through this barrier and are able to model the probabilities of sequences drawn from the source near-optimally, achieving small cross-entropy loss. With this observation as starting point, we study the end-to-end cross-entropy loss achieved by transformers with and without tokenization. With the appropriate tokenization, we show that even the simplest unigram models (over tokens) learnt by transformers are able to model the probability of sequences drawn from $k^{\text{th}}$-order Markov sources near optimally. Our analysis provides a justification for the use of tokenization in practice through studying the behavior of transformers on Markovian data.