Can Perplexity Predict Fine-tuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali

TL;DR

This study investigates how different tokenization strategies affect Nepali language models' understanding capabilities, revealing that SentencePiece tokenization outperforms byte-level BPE in downstream tasks for non-Latin scripts.

Contribution

The paper provides a comprehensive evaluation of six tokenization schemes on Nepali transformer models, emphasizing the importance of tokenization choices beyond perplexity for low-resource languages.

Findings

SentencePiece tokenization improves understanding tasks in Nepali models

Byte-level BPE is less effective for Nepali language understanding

Insights for developing better language models in low-resource, non-Latin script languages

Abstract

The impact of subword tokenization on language model performance is well-documented for perplexity, with finer granularity consistently reducing this intrinsic metric. However, research on how different tokenization schemes affect a model's understanding capabilities remains limited, particularly for non-Latin script languages. Addressing this gap, we conducted a comprehensive evaluation of six distinct tokenization strategies by pretraining transformer-based language models for Nepali and evaluating their performance across multiple downstream tasks. While recent prominent models like GPT, RoBERTa, Claude, LLaMA, Mistral, Falcon, and MPT have adopted byte-level BPE tokenization, our findings demonstrate that for Nepali, SentencePiece tokenization consistently yields superior results on understanding-based tasks. Unlike previous studies that primarily focused on BERT-based architectures, our research specifically examines sequential transformer models, providing valuable insights for language model development in low-resource languages and highlighting the importance of tokenization strategy beyond perplexity reduction.