Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language

TL;DR

This paper benchmarks various BERT-based models for sentence-level topic classification in Nepali, revealing that Indic models like MuRIL-large outperform others and setting a baseline for future Nepali NLP tasks.

Contribution

It provides the first comprehensive evaluation of multiple BERT variants for Nepali sentence classification, highlighting the effectiveness of Indic models in low-resource language NLP.

Findings

MuRIL-large achieved the highest F1-score of 90.60%.

NepBERTa performed competitively with an F1-score of 88.26%.

Indic models outperform multilingual and monolingual models in Nepali classification.

Abstract

Transformer-based models such as BERT have significantly advanced Natural Language Processing (NLP) across many languages. However, Nepali, a low-resource language written in Devanagari script, remains relatively underexplored. This study benchmarks multilingual, Indic, Hindi, and Nepali BERT variants to evaluate their effectiveness in Nepali topic classification. Ten pre-trained models, including mBERT, XLM-R, MuRIL, DevBERT, HindiBERT, IndicBERT, and NepBERTa, were fine-tuned and tested on the balanced Nepali dataset containing 25,006 sentences across five conceptual domains and the performance was evaluated using accuracy, weighted precision, recall, F1-score, and AUROC metrics. The results reveal that Indic models, particularly MuRIL-large, achieved the highest F1-score of 90.60%, outperforming multilingual and monolingual models. NepBERTa also performed competitively with an F1-score of 88.26%. Overall, these findings establish a robust baseline for future document-level classification and broader Nepali NLP applications.