Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models

TL;DR

This paper investigates alternative BPE inference algorithms that do not rely on the merge list, finding they often preserve model performance and could enhance privacy in language model tokenization.

Contribution

It introduces and evaluates merge-list-free BPE inference methods, demonstrating minimal performance loss and potential privacy benefits compared to traditional merge-list-based approaches.

Findings

Non-targeted BPE inference algorithms maintain performance across tasks.

Targeted deviations from merge lists cause significant performance degradation.

Merge-list-free schemes offer a promising privacy-preserving alternative.

Abstract

Standard Byte-Pair Encoding (BPE) tokenization compresses text by pairing a learned token vocabulary with a detailed merge list. Recent work has shown that this merge list exposes a potential attack surface for extracting information about language model's training data. In this paper, we explore the downstream impact of BPE inference algorithms that do not rely on this merge list at all, and hence differ from the encoding process during BPE training. To address this question, we investigate two broad classes of BPE inference schemes that differ from BPE application during training: a) targeted deviation from merge-lists including random merge orders, and various corruptions of merge list involving deletion/truncation, and b) non-targeted BPE inference algorithms that do not depend on the merge list but focus on compressing the text either greedily or exactly. Extensive experiments across diverse language modeling tasks like accuracy-based QA benchmarks, machine translation, and open-ended generation reveal that while targeted deviation from the merge lists exhibits significant degradation in language model performance, the non-targeted merge-list-free inference algorithms result in minimal impact on downstream performance that is often much smaller than expected. These findings pave way for simpler and potentially more privacy-preserving tokenization schemes that do not catastrophically compromise model performance.