Explaining and Mitigating Crosslingual Tokenizer Inequities

TL;DR

This paper investigates the causes of crosslingual tokenizer inequities, demonstrating that vocabulary size and pre-tokenization significantly influence token premiums, and proposes methods to mitigate these disparities.

Contribution

It systematically analyzes factors affecting token premiums across languages and introduces strategies like optimal vocabulary sizing and superword tokenizers to reduce inequities.

Findings

Vocabulary size impacts token premiums but increasing it alone does not reduce disparities.

Optimal vocabulary sizes tailored to each language significantly decrease token premiums.

Superword tokenizers reduce token premiums and improve compression across languages.

Abstract

The number of tokens it takes to encode parallel text in different languages is known to vary. These disparities are called token premiums. Having high token premiums leads to less throughput during training and increases costs at inference. In this paper, we show that even after controlling for dataset size, vocabulary size, and data content, monolingual tokenizers exhibit a wide range of token premiums across languages. To understand the cross-linguistic differences that cause these token premiums, we train a suite of approximately 7,000 comparable monolingual tokenizers for 97 languages, manipulating tokenization algorithm, vocabulary size, and dataset size. We measure token premiums and test for a relationship between factors such as data similarity (between tokenizer training and evaluation), vocabulary size, and pre-tokenization. We also investigate the role of language-specific features such as writing system and word length. We find that similarity between training and test data does not impact token premiums, but vocabulary size and pre-tokenization do. While simply increasing vocabulary size does not lead to reduced token premium effects, we can determine an ``optimal'' vocabulary size for each language to achieve significantly reduced token premium effects. We also train superword tokenizers which allow merges over whitespaces, and we find that they both reduce token premium effects and improve compression overall. Thus, intervening on the vocabulary size or the pre-tokenizer significantly reduces crosslingual token premium effects.