Language Models over Canonical Byte-Pair Encodings

TL;DR

This paper addresses the issue of language models assigning probability to noncanonical token encodings, proposing methods to enforce canonicality and improve model likelihoods.

Contribution

It introduces two approaches—conditioning and construction—to ensure language models only produce canonical token strings, reducing errors and improving performance.

Findings

Enforcing canonicality improves held-out data likelihood.

Conditioning approach works without additional training.

Construction approach guarantees canonical outputs during training.

Abstract

Modern language models represent probability distributions over character strings as distributions over (shorter) token strings derived via a deterministic tokenizer, such as byte-pair encoding. While this approach is highly effective at scaling up language models to large corpora, its current incarnations have a concerning property: the model assigns nonzero probability mass to an exponential number of $\it{noncanonical}$ token encodings of each character string -- these are token strings that decode to valid character strings but are impossible under the deterministic tokenizer (i.e., they will never be seen in any training corpus, no matter how large). This misallocation is both erroneous, as noncanonical strings never appear in training data, and wasteful, diverting probability mass away from plausible outputs. These are avoidable mistakes! In this work, we propose methods to enforce canonicality in token-level language models, ensuring that only canonical token strings are assigned positive probability. We present two approaches: (1) canonicality by conditioning, leveraging test-time inference strategies without additional training, and (2) canonicality by construction, a model parameterization that guarantees canonical outputs but requires training. We demonstrate that fixing canonicality mistakes improves the likelihood of held-out data for several models and corpora.