Batching BPE Tokenization Merges

TL;DR

This paper introduces BatchBPE, a Python implementation that enables efficient batching of BPE merges, making high-quality tokenizer training feasible on low-resource hardware and facilitating experimentation with new tokenization strategies.

Contribution

The paper presents BatchBPE, a novel approach and open-source tool that allows batching in BPE tokenization, reducing memory usage and enabling tokenizer training on basic laptops.

Findings

BatchBPE effectively merges hundreds of token pairs simultaneously.

It reduces memory footprint during vocabulary training.

The tool supports experimentation with preprocessing and merge strategies.

Abstract

The Byte Pair Encoding algorithm can be safely batched to merge hundreds of pairs of tokens at a time when building up a tokenizer's vocabulary. This technique combined with reducing the memory footprint of text used in vocabulary training make it feasible to train a high quality tokenizer on a basic laptop. This paper presents BatchBPE, an open-source pure Python implementation of these concepts, with the goal of making experimenting with new tokenization strategies more accessible especially in compute- and memory-constrained contexts. BatchBPE's usefulness and malleability are demonstrated through the training of several token vocabularies to explore the batch merging process and experiment with preprocessing a stop word list and ignoring the least common text chunks in a dataset. Resultant encoded lengths of texts are used as a basic evaluation metric.