Do\u{g}al Dil \.I\c{s}lemede Tokenizasyon Standartlar{\i} ve \"Ol\c{c}\"um\"u: T\"urk\c{c}e \"Uzerinden B\"uy\"uk Dil Modellerinin Kar\c{s}{\i}la\c{s}t{\i}rmal{\i} Analizi

TL;DR

This paper presents a new evaluation framework for tokenization in morphologically-rich languages like Turkish, emphasizing language-specific metrics and demonstrating their impact on large language model performance.

Contribution

It introduces a novel set of metrics and a framework for assessing tokenization quality tailored to morphologically complex, low-resource languages.

Findings

Language-specific token percentages correlate strongly with downstream task performance.

Increasing model size alone does not improve linguistic understanding.

Proposed standards improve tokenization effectiveness for Turkish.

Abstract

Tokenization is a fundamental preprocessing step in Natural Language Processing (NLP), significantly impacting the capability of large language models (LLMs) to capture linguistic and semantic nuances. This study introduces a novel evaluation framework addressing tokenization challenges specific to morphologically-rich and low-resource languages such as Turkish. Utilizing the Turkish MMLU (TR-MMLU) dataset, comprising 6,200 multiple-choice questions from the Turkish education system, we assessed tokenizers based on vocabulary size, token count, processing time, language-specific token percentages (\%TR), and token purity (\%Pure). These newly proposed metrics measure how effectively tokenizers preserve linguistic structures. Our analysis reveals that language-specific token percentages exhibit a stronger correlation with downstream performance (e.g., MMLU scores) than token purity. Furthermore, increasing model parameters alone does not necessarily enhance linguistic performance, underscoring the importance of tailored, language-specific tokenization methods. The proposed framework establishes robust and practical tokenization standards for morphologically complex languages.