Towards Nepali-language LLMs: Efficient GPT training with a Nepali BPE tokenizer

TL;DR

This paper introduces a Nepali-language GPT-2 model trained with a custom BPE tokenizer and optimized training strategies, demonstrating effective Nepali text generation despite limited resources.

Contribution

It presents a novel Nepali-specific BPE tokenizer and training methodology for GPT-2, tailored for low-resource Nepali NLP tasks.

Findings

Achieved a perplexity of 21.80 on Nepali text

Demonstrated coherent Nepali news-style text generation

Implemented memory-efficient training with FlashAttention

Abstract

Nepali, a low-resource language spoken by over 32 million people, continues to face challenges in natural language processing (NLP) due to its complex grammar, agglutinative morphology, and limited availability of high-quality corpora. Most efforts to date have centered on basic encoder architectures; they remain insufficient for Nepali-specific text generation. This study presents a GPT-2-based Nepali language model trained using several training strategies inspired by GPT-3, including optimized learning rate schedules, batch scaling, and architectural refinements. A custom 16k Byte-Pair Encoding (BPE) tokenizer was trained exclusively on Nepali text to ensure more consistent segmentation and improved input representation. The model was pretrained on a combined dataset comprising a 10.75GB cleaned NepBERTa corpus and additional web-scraped Nepali news articles. FlashAttention was integrated to reduce memory usage and stabilize training. After two epochs, the model achieved a training loss of 3.168177, a validation loss of 3.081982, and a final perplexity of 21.80, demonstrating its capability to generate coherent Nepali news-style text.