The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models

TL;DR

This paper quantifies how script differences impact tokenization efficiency and latency in multilingual models, revealing significant disparities that suggest the need for script-aware tokenization strategies.

Contribution

It introduces a systematic measurement of script-induced costs in multilingual models, highlighting tokenization as a source of inequity and proposing the importance of script-aware methods.

Findings

Orthographic variants cause up to 16.5x inference slowdown.

Higher fragmentation increases information cost by up to 47%.

Tokenization disparities reflect orthography-conditioned processing.

Abstract

Pretrained multilingual language models are often assumed to be script-agnostic, yet their tokenizers can impose systematic costs on certain writing systems. We quantify this script tax by comparing two orthographic variants with identical linguistic content. Across mBERT and XLM-R, the higher-fragmentation orthography shows a ~3.4x increase in fertility (6.73-6.85 vs. 2.10-2.35 tokens/word), leading to a 16.5x inference slowdown (0.23 vs. 3.8 sentences/second) on identical hardware. Using bits per character (BPC) to avoid the "NLL paradox" from subword fragmentation, we find a substantial increase in information cost: +19.7% for mBERT (8.06->9.65) and +47.1% for XLM-R (12.19->17.94). A round-trip conversion check (CER_rt=0.31) suggests these gaps reflect orthography-conditioned processing rather than mapping noise. Our results highlight tokenization as a key source of inequity in multilingual NLP and motivate script-aware tokenization and pretraining.