Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations

TL;DR

This paper reveals that language models are surprisingly robust to non-canonical tokenizations, maintaining high performance across benchmarks and even benefiting from certain non-standard tokenization schemes, with robustness stemming from instruction-tuning.

Contribution

It demonstrates the robustness of instruction-tuned language models to unseen non-canonical tokenizations and explores how tokenization can be optimized to improve specific tasks.

Findings

Models retain up to 93.4% performance with random tokenizations.

Character-level tokenization improves string and code tasks by up to +14%.

Non-canonical tokenization robustness arises during instruction-tuning.

Abstract

Modern tokenizers employ deterministic algorithms to map text into a single "canonical" token sequence, yet the same string can be encoded as many non-canonical tokenizations using the tokenizer vocabulary. In this work, we investigate the robustness of LMs to text encoded with non-canonical tokenizations entirely unseen during training. Surprisingly, when evaluated across 20 benchmarks, we find that instruction-tuned models retain up to 93.4% of their original performance when given a randomly sampled tokenization, and 90.8% with character-level tokenization. We see that overall stronger models tend to be more robust, and robustness diminishes as the tokenization departs farther from the canonical form. Motivated by these results, we then identify settings where non-canonical tokenization schemes can *improve* performance, finding that character-level segmentation improves string manipulation and code understanding tasks by up to +14%, and right-aligned digit grouping enhances large-number arithmetic by +33%. Finally, we investigate the source of this robustness, finding that it arises in the instruction-tuning phase. We show that while both base and post-trained models grasp the semantics of non-canonical tokenizations (perceiving them as containing misspellings), base models try to mimic the imagined mistakes and degenerate into nonsensical output, while post-trained models are committed to fluent responses. Overall, our findings suggest that models are less tied to their tokenizer than previously believed, and demonstrate the promise of intervening on tokenization at inference time to boost performance.