Reducing Tokenization Premiums for Low-Resource Languages

TL;DR

This paper investigates the high tokenization costs faced by low-resource languages in modern language models and proposes a post-hoc vocabulary extension method to reduce these premiums, improving efficiency and context utilization.

Contribution

It introduces a novel post-hoc vocabulary extension technique to reduce tokenization premiums for low-resource languages in pre-trained models.

Findings

Reduced tokenization premiums in 12 low-resource languages

Similar last hidden states between original and compressed inputs

Potential for lower costs and improved context in language models

Abstract

Relative to English, low-resource languages suffer from substantial tokenization premiums in modern LMs, meaning that it generally requires several times as many tokens to encode a sentence in a low-resource language than to encode the analogous sentence in English. This tokenization premium results in increased API and energy costs and reduced effective context windows for these languages. In this paper we analyze the tokenizers of ten popular LMs to better understand their designs and per-language tokenization premiums. We also propose a mechanism to reduce tokenization premiums in pre-trained models, by post-hoc additions to the token vocabulary that coalesce multi-token characters into single tokens. We apply this methodology to 12 low-resource languages, demonstrating that the original and compressed inputs often have similar last hidden states when run through the Llama 3.2 1B model.