The Foundations of Tokenization: Statistical and Computational Concerns

TL;DR

This paper develops a formal framework for understanding tokenization in NLP, addressing its theoretical foundations, statistical properties, and computational concerns to improve language model robustness.

Contribution

It introduces a unified formal framework based on stochastic maps to analyze tokenizers and establish conditions for their consistency and reliability.

Findings

Provides necessary and sufficient conditions for tokenizer consistency

Analyzes statistical and computational issues like ambiguity and finiteness

Lays groundwork for more robust neural language representations

Abstract

Tokenization - the practice of converting strings of characters from an alphabet into sequences of tokens over a vocabulary - is a critical step in the NLP pipeline. The use of token representations is widely credited with increased model performance but is also the source of many undesirable behaviors, such as spurious ambiguity or inconsistency. Despite its recognized importance as a standard representation method in NLP, the theoretical underpinnings of tokenization are not yet fully understood. In particular, the impact of tokenization on language model estimation has been investigated primarily through empirical means. The present paper contributes to addressing this theoretical gap by proposing a unified formal framework for representing and analyzing tokenizer models. Based on the category of stochastic maps, this framework enables us to establish general conditions for a principled use of tokenizers and, most importantly, the necessary and sufficient conditions for a tokenizer model to preserve the consistency of statistical estimators. In addition, we discuss statistical and computational concerns crucial for designing and implementing tokenizer models, such as inconsistency, ambiguity, finiteness, and sequentiality. The framework and results advanced in this paper contribute to building robust theoretical foundations for representations in neural language modeling that can inform future theoretical and empirical research.