LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model

TL;DR

LoPT introduces a lossless parallel tokenization method that significantly speeds up long context inference in large language models without sacrificing accuracy, by ensuring identical output to sequential tokenization.

Contribution

We propose LoPT, a novel framework that achieves lossless parallel tokenization with boundary-aware merging, addressing the bottleneck in long-sequence processing for LLMs.

Findings

LoPT achieves substantial speedup in tokenization time.

LoPT guarantees identical output to sequential tokenization.

Theoretical proof confirms consistency and robustness.

Abstract

Long context inference scenarios have become increasingly important for large language models, yet they introduce significant computational latency. While prior research has optimized long-sequence inference through operators, model architectures, and system frameworks, tokenization remains an overlooked bottleneck. Existing parallel tokenization methods accelerate processing through text segmentation and multi-process tokenization, but they suffer from inconsistent results due to boundary artifacts that occur after merging. To address this, we propose LoPT, a novel Lossless Parallel Tokenization framework that ensures output identical to standard sequential tokenization. Our approach employs character-position-based matching and dynamic chunk length adjustment to align and merge tokenized segments accurately. Extensive experiments across diverse long-text datasets demonstrate that LoPT achieves significant speedup while guaranteeing lossless tokenization. We also provide theoretical proof of consistency and comprehensive analytical studies to validate the robustness of our method.