Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs

TL;DR

This paper evaluates how well large language models and their tokenizers represent Arabic root-pattern morphology, revealing that tokenization alignment does not necessarily improve morphological generation or downstream tasks.

Contribution

It provides a comprehensive analysis of LLMs' handling of Arabic morphology and questions the importance of morphological tokenization for model performance.

Findings

Tokenizer morphological alignment is neither necessary nor sufficient for morphological generation.

LLMs can generate productive root-pattern forms without explicit morphological tokenization.

Morphological fidelity varies across different tokenizers and models.

Abstract

This work investigates how effectively large language models (LLMs) and their tokenization schemes represent and generate Arabic root-pattern morphology, probing whether they capture genuine morphological structure or rely on surface memorization. Arabic morphological system provides a rich testbed for analyzing how LLMs handle complex, non-concatenative forms and how tokenization choices influence this process. Our study begins with an evaluation of morphological fidelity across Arabic and multilingual tokenizers against gold-standard segmentation, followed by an analysis of LLM performance in productive root-pattern generation using a newly developed test set. Our findings across seven Arabic-centric and multilingual LLMs and their respective tokenizers reveal that tokenizer morphological alignment is not necessary nor sufficient for morphological generation, which questions the role of morphological tokenization in downstream performance.