Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE

TL;DR

This paper introduces BengaliBPE, a language-specific subword tokenizer for Bengali that improves morphological consistency and interpretability over existing methods, demonstrated through extensive evaluation on a Bengali news classification dataset.

Contribution

BengaliBPE is a novel, morphology-aware BPE tokenizer tailored for Bengali, addressing limitations of existing tokenizers designed for Latin scripts.

Findings

BengaliBPE achieves more detailed segmentation and better morphological interpretability.

BengaliBPE has slightly higher computational cost but improves downstream classification accuracy.

Language-aware tokenization is crucial for morphologically rich scripts like Bengali.

Abstract

Tokenization is an important first step in Natural Language Processing (NLP) pipelines because it decides how models learn and represent linguistic information. However, current subword tokenizers like SentencePiece or HuggingFace BPE are mostly designed for Latin or multilingual corpora and do not perform well on languages with rich morphology such as Bengali. To address this limitation, we present BengaliBPE, a Byte Pair Encoding (BPE) tokenizer specifically developed for the Bengali script. BengaliBPE applies Unicode normalization, grapheme-level initialization, and morphology-aware merge rules to maintain linguistic consistency and preserve subword integrity. We use a large-scale Bengali news classification dataset to compare BengaliBPE with three baselines: Whitespace, SentencePiece BPE, and HuggingFace BPE. The evaluation considers tokenization granularity, encoding speed, and downstream classification accuracy. While all methods perform reasonably well, BengaliBPE provides the most detailed segmentation and the best morphological interpretability, albeit with slightly higher computational cost. These findings highlight the importance of language-aware tokenization for morphologically rich scripts and establish BengaliBPE as a strong foundation for future Bengali NLP systems, including large-scale pretraining of contextual language models.