Say Anything but This: When Tokenizer Betrays Reasoning in LLMs

TL;DR

This paper reveals that subword tokenizers can cause LLM reasoning failures due to representational mismatches, leading to phantom edits and systematic artifacts that undermine model reasoning accuracy.

Contribution

The study introduces a tokenizer consistency probe and provides a taxonomy of eight systematic tokenizer artifacts affecting LLM reasoning.

Findings

Over 11,000 replacement trials analyzed across state-of-the-art LLMs.

Non-trivial rate of phantom edits caused by tokenizer artifacts.

Identification of systematic tokenizer artifacts like whitespace shifts and intra-word resegmentation.

Abstract

Large language models (LLMs) reason over discrete token ID sequences, yet modern subword tokenizers routinely produce non-unique encodings: multiple token ID sequences can detokenize to identical surface strings. This representational mismatch creates an unmeasured fragility wherein reasoning processes can fail. LLMs may treat two internal representations as distinct "words" even when they are semantically identical at the text level. In this work, we show that tokenization can betray LLM reasoning through one-to-many token ID mappings. We introduce a tokenization-consistency probe that requires models to replace designated target words in context while leaving all other content unchanged. The task is intentionally simple at the surface level, enabling us to attribute failures to tokenizer-detokenizer artifacts rather than to knowledge gaps or parameter limitations. Through analysis of over 11000 replacement trials across state-of-the-art open-source LLMs, we find a non-trivial rate of outputs exhibit phantom edits: cases where models operate under the illusion of correct reasoning, a phenomenon arising from tokenizer-induced representational defects. We further analyze these cases and provide a taxonomy of eight systematic tokenizer artifacts, including whitespace-boundary shifts and intra-word resegmentation. These findings indicate that part of apparent reasoning deficiency originates in the tokenizer layer, motivating tokenizer-level remedies before incurring the cost of training ever-larger models on ever-larger corpora.