Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization

TL;DR

This paper introduces Parity-aware Byte Pair Encoding, a novel tokenization method that enhances cross-lingual fairness by balancing tokenization quality across languages without sacrificing overall compression or downstream performance.

Contribution

It proposes a new BPE variant that prioritizes equitable tokenization for low-resource languages, addressing biases in standard frequency-based algorithms.

Findings

More equitable token counts across languages

Negligible impact on overall compression rate

No substantial effect on downstream task performance

Abstract

Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines. Standard algorithms for learning tokenizers rely on frequency-based objectives, which favor languages dominant in the training data and consequently leave lower-resource languages with tokenizations that are disproportionately longer, morphologically implausible, or even riddled with <UNK> placeholders. This phenomenon ultimately amplifies computational and financial inequalities between users from different language backgrounds. To remedy this, we introduce Parity-aware Byte Pair Encoding (BPE), a variant of the widely-used BPE algorithm. At every merge step, Parity-aware BPE maximizes the compression gain of the currently worst-compressed language, trading a small amount of global compression for cross-lingual parity. We find empirically that Parity-aware BPE leads to more equitable token counts across languages, with negligible impact on global compression rate and no substantial effect on language-model performance in downstream tasks.