Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment

TL;DR

This study evaluates how different tokenization algorithms, especially Unigram and BPE, affect NLP performance in morphologically rich languages like Telugu, revealing Unigram's superiority and the importance of morphological alignment.

Contribution

It provides a comprehensive comparison of tokenizer algorithms for Telugu, highlighting Unigram's dominance and the limited impact of morphological alignment on downstream tasks.

Findings

Unigram tokenizers outperform BPE in Telugu.

Morphological alignment correlates moderately with performance.

Hybrid approaches with morphological pre-segmentation improve BPE results.

Abstract

The relationship between tokenizer algorithm (e.g., Byte-Pair Encoding (BPE), Unigram), morphological alignment, tokenization quality (e.g., compression efficiency), and downstream performance remains largely unclear, particularly for languages with complex morphology. In this paper, we conduct a comprehensive evaluation of tokenizers using small-sized BERT models -- from pre-training through fine-tuning -- for Telugu (agglutinative), along with preliminary evaluation in Hindi (primarily fusional with some agglutination) and English (fusional). To evaluate morphological alignment of tokenizers in Telugu, we create a dataset containing gold morpheme segmentations of 600 derivational and 7000 inflectional word forms. Our experiments reveal two key findings for Telugu. First, the choice of tokenizer algorithm is the most significant factor influencing performance, with Unigram-based tokenizers consistently outperforming BPE across most settings. Second, while better morphological alignment shows a moderate, positive correlation with performance on text classification and structure prediction tasks, its impact is secondary to the tokenizer algorithm. Notably, hybrid approaches that use morphological information for pre-segmentation significantly boost the performance of BPE, though not Unigram. Our results further showcase the need for comprehensive intrinsic evaluation metrics for tokenizers that could explain downstream performance trends consistently.