SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance

TL;DR

SupraTok introduces a novel tokenization method that learns multi-word semantic units, improving efficiency and performance across multiple languages and benchmarks, thereby enhancing language model capabilities.

Contribution

The paper presents SupraTok, a new tokenization architecture that extends Byte-Pair Encoding with cross-boundary pattern learning and curriculum strategies, achieving significant efficiency gains and improved downstream task performance.

Findings

31% improvement in English tokenization efficiency

8.4% performance boost on HellaSWAG

9.5% improvement on MMLU benchmarks

Abstract

Tokenization remains a fundamental yet underexplored bottleneck in natural language processing, with strategies largely static despite remarkable progress in model architectures. We present SupraTok, a novel tokenization architecture that reimagines subword segmentation through three innovations: cross-boundary pattern learning that discovers multi-word semantic units, entropy-driven data curation that optimizes training corpus quality, and multi-phase curriculum learning for stable convergence. Our approach extends Byte-Pair Encoding by learning "superword" tokens, coherent multi-word expressions that preserve semantic unity while maximizing compression efficiency. SupraTok achieves 31% improvement in English tokenization efficiency (5.91 versus 4.51 characters per token) compared to OpenAI's o200k tokenizer and 30% improvement over Google's Gemma 3 tokenizer (256k vocabulary), while maintaining competitive performance across 38 languages. When integrated with a GPT-2 scale model (124M parameters) trained on 10 billion tokens from the FineWeb-Edu dataset, SupraTok yields 8.4% improvement on HellaSWAG and 9.5% on MMLU benchmarks without architectural modifications. While these results are promising at this scale, further validation at larger model scales is needed. These findings suggest that efficient tokenization can complement architectural innovations as a path to improved language model performance.