Are you going to finish that? A Practical Study of the Partial Token Problem

TL;DR

This paper investigates the partial token problem in language models, revealing its prevalence in realistic prompts across languages and code, and evaluates mitigation strategies to address the resulting probability distortions.

Contribution

It systematically studies the partial token problem in natural and code prompts, quantifies its severity, and evaluates practical mitigation techniques for inference-time correction.

Findings

Partial tokens cause significant probability distortion in language models.

The problem persists and worsens with larger models.

Recent exact mitigation solutions are effective in reducing probability errors.

Abstract

Language models (LMs) are trained over sequences of tokens, whereas users interact with LMs via text. This mismatch gives rise to the partial token problem, which occurs when a user ends their prompt in the middle of the expected next-token, leading to distorted next-token predictions. Although this issue has been studied using arbitrary character prefixes, its prevalence and severity in realistic prompts respecting word boundaries remains underexplored. In this work, we identify three domains where token and "word" boundaries often do not line up: languages that do not use whitespace, highly compounding languages, and code. In Chinese, for example, up to 25% of word boundaries do not line up with token boundaries, making even natural, word-complete prompts susceptible to this problem. We systematically construct semantically natural prompts ending with a partial tokens; in experiments, we find that they comprise a serious failure mode: frontier LMs consistently place three orders of magnitude less probability on the correct continuation compared to when the prompt is "backed-off" to be token-aligned. This degradation does not diminish with scale and often worsens for larger models. Finally, we evaluate inference-time mitigations to the partial token problem and validate the effectiveness of recent exact solutions. Overall, we demonstrate the scale and severity of probability distortion caused by tokenization in realistic use cases, and provide practical recommentions for model inference providers.