SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling

TL;DR

SemToken introduces a semantic-aware tokenization method that reduces token redundancy and enhances efficiency in long-context language modeling without sacrificing accuracy.

Contribution

It proposes a novel semantic-aware tokenization framework that dynamically adjusts token granularity based on semantic density, improving efficiency over traditional frequency-based methods.

Findings

Achieves up to 2.4x reduction in token count

Realizes up to 1.9x speedup in language modeling tasks

Maintains comparable perplexity and accuracy with baseline models

Abstract

Tokenization plays a critical role in language modeling, yet existing approaches such as Byte-Pair Encoding (BPE) or WordPiece operate purely on frequency statistics, ignoring the underlying semantic structure of text. This leads to over-tokenization of semantically redundant spans and underutilization of contextual coherence, particularly in long-context scenarios. In this work, we propose \textbf{SemToken}, a semantic-aware tokenization framework that jointly reduces token redundancy and improves computation efficiency. SemToken first extracts contextual semantic embeddings via lightweight encoders and performs local semantic clustering to merge semantically equivalent tokens. Then, it allocates heterogeneous token granularity based on semantic density, allowing finer-grained tokenization in content-rich regions and coarser compression in repetitive or low-entropy spans. SemToken can be seamlessly integrated with modern language models and attention acceleration methods. Experiments on long-context language modeling benchmarks such as WikiText-103 and LongBench show that SemToken achieves up to $2.4\times$ reduction in token count and $1.9\times$ speedup, with negligible or no degradation in perplexity and downstream accuracy. Our findings suggest that semantic structure offers a promising new axis for optimizing tokenization and computation in large language models.